mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
8299f7c38956db97e6a4a538770bd2ecbae13cda
8126
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
8299f7c389 |
perf(runner): check a listening Runner's backlog every 10 ms, not every reconnect delay (#21821)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A remote `paperclip_runner` agent runs `paperclip-runnerd` in a sandbox or VM. A provider ingress proxies Core's WebSocket to a Runner that listens (`--listen-address`). > - In listen mode, the Runner checks for an incoming connection only once per `--reconnect-delay-ms` (default 250 ms). It waits that delay again after each disconnect, and it backs off after a failed handshake. > - That delay exists to pace redials in dial mode. A listener never dials. Core's connection waits in the kernel backlog until the next `accept`. > - Every listen-mode connect pays part of that wait. A warm run attachment pays it in full: the Runner disconnects at the authority rotation, and Core's new-authority connect then waits up to 250 ms or more. > - This pull request makes a listening Runner check for a connection every 10 ms. Dial mode keeps the configured reconnect delay. > - The benefit is about 0.4 s less on each warm run attachment and about 0.1 s less on each cold start, for every listen-mode Runner. ## Linked Issues or Issue Description **What happened?** A warm follow-up turn on a listen-mode remote Runner reached its first turn later than a cold turn on the same agent. In traces, a cold start makes one Core→Runner connect of 134–178 ms. Each warm attachment makes an old-authority reconnect (72–233 ms) and then a new-authority connect of 411–504 ms. This was 6 of 6 warm runs. **Expected behavior** A connect to a listening Runner completes in about one network round trip plus the WebSocket handshake. It does not wait for a reconnect timer. **Steps to reproduce** 1. Start `paperclip-runnerd` in listen mode with the default `--reconnect-delay-ms`. 2. Let it reach the disconnected wait (startup, a park claim, or a `run.attach` authority rotation). 3. Connect to the listener. The upgrade is not answered until the next 250 ms check. **Root cause** `durable/runner.rs` schedules the next `accept` at `now + config.reconnect_delay` when no connection is pending, after a disconnect, and as the base of the failure backoff. The Launcher-style hosts do not pass `--reconnect-delay-ms`, so the 250 ms default applies. The tests use a 1 ms delay, so they did not show the wait. ## What Changed - `durable/transport.rs`: add `RunnerTransportEndpoint::retry_delay()`. A listener returns `min(configured, 10 ms)`. A dial endpoint returns the configured delay without change. - `durable/runner.rs`: the four places that waited `config.reconnect_delay` (no pending connection, failure backoff base, two disconnect paths) now use `endpoint.retry_delay(config.reconnect_delay)`. - `DURABLE_TRANSPORT.md`: document the listen-mode connection check interval. - Add a unit test for the listener and dial delays. - Add `tests/listener_accept.rs`. It starts `paperclip-runnerd` in listen mode with the default reconnect delay. One connection drops before authentication, as a superseded route's redial does. The test then checks that the next upgrade is answered in under 120 ms. Before this change, that upgrade waited 188–313 ms. ## Verification - `cargo test -p paperclip-runner-core --lib` passes locally. - `cargo clippy -p paperclip-runner-core --all-targets` reports no new findings in the changed files. - `cargo fmt --all -- --check` passes. - New tests: `durable::transport::tests::listener_checks_its_backlog_sooner_than_a_dial_reconnects` and `tests/listener_accept.rs`. - `tests/listener_accept.rs` fails on the previous `runner.rs` and passes with this change (3 runs each). - Existing listener and warm-attachment tests pass (`listener_*`, `warm_run_attachment_*`, `warm_external_attachment_*`). - I did not measure the change on a live host. The effect is visible in the next deployment as a shorter `runner.transport.registration` on warm attachments. ## Risks - Low risk. Dial mode does not change. - A disconnected listener now wakes every 10 ms instead of every 250 ms. Each wake is one non-blocking `accept` and a non-blocking host-control read. The loop already slept in 25 ms steps. - Failed handshakes on a listener now back off from 10 ms (10, 20, 40 … 320 ms) instead of from 250 ms. Core's own reconnect loop still paces its redials, and the ingress requires credentials, so this does not create a hot loop. - The change needs a Runner release before hosts pick it up. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, trace queries). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>canary/v2026.1115.0-canary.20 |
||
|
|
304e40fe9c |
fix(auth): report effective Cloud owner access to the board UI (#21817)
## Thinking Path > - Paperclip manages AI agents and the environments where they run. > - Cloud owners can receive instance-admin rights for their own dedicated instance through the existing owner setting. > - Authentication computes these rights for each request without storing an instance-admin role. > - The board UI reads `/api/cli-auth/me` to decide whether to show administration controls such as Tailscale. > - That response used only stored roles, hiding controls that the Cloud backend already authorized. > - This change reports the effective Cloud request rights so the UI agrees with the backend. ## Linked Issues or Issue Description **What happened?** A Cloud owner with `enableOwnerInstanceAdmin` enabled can successfully read Tailscale connections, while `/api/cli-auth/me` reports `isInstanceAdmin: false`. The Connectors page therefore hides Tailscale. The same stale summary can hide other instance administration controls. **Expected behavior** The access summary should reflect the Cloud request's effective privileges. Turning off owner elevation must remove those privileges on the next request. Session and board-key callers must continue to use the fresh stored-role snapshot. **Steps to reproduce** Sign in as a Cloud stack owner with owner instance-admin elevation enabled and no stored instance-admin role. Open Connectors and compare the access summary with a Tailscale connections request: the summary reports no admin access even though the backend permits the request. Searched existing PRs for `"cli-auth/me" cloud` and `tailscale access`; no duplicate fix was found. Related: [#21526](https://github.com/paperclipai/paperclip/pull/21526), which added device sign-in and exposed this hosted permission-summary gap. ## What Changed - `/api/cli-auth/me` uses the trusted Cloud actor's computed `isInstanceAdmin` value for `cloud_tenant` requests. - Session and board-key requests retain fresh stored-role checks. No roles are persisted and no authorization gates or owner settings change. - Regression tests cover Cloud elevation and de-elevation, non-Cloud stored access, and rejection of anonymous/agent callers. Tailscale documentation explains Cloud owner access. ## Verification - Reproduced the deployed mismatch in an authenticated Cloud owner browser session: the access summary reported false while the Tailscale list request succeeded. No feature connection or configuration was changed. - Both Cloud access regression cases failed before the fix. - `pnpm exec vitest run server/src/__tests__/cli-auth-routes.test.ts server/src/middleware/cloud-tenant-actor.test.ts ui/src/pages/apps/TailscaleConnector.test.tsx` passes: 88 tests. - `pnpm --filter @paperclipai/server typecheck` passes. - `pnpm verify --explain` exits 2 and selects broad CI. All 57 checks pass on final head `6044d5097396de95f49bcbb7ce7bbb1e445a80fa`; no checks remain pending. Greptile scored this head 5/5 with no unresolved comments, and the branch is conflict-free. - `git diff --check` and a local credential/PII scan pass. - The fix has not been deployed or accepted in a hosted browser yet. ## Risks The summary exposes administration controls to Cloud owners already authorized by the backend. Non-owner access and disabled elevation remain denied by the existing authentication policy. No migration, provider configuration, or deployment change. ## Model Used OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) through Codex in T3 Code, with xhigh reasoning. Context window size was not exposed. No subagents. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in-PR following the bug template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal ticket or instance details - [x] I have run tests locally and they pass - [x] I have added regression tests - [x] I have updated relevant documentation - [x] I have considered and documented risks - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no unresolved comments - [x] I will address actionable reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>canary/v2026.1115.0-canary.19 |
||
|
|
f71aa135a0 |
fix(server): release sandbox leases for runs stopped before runner attach (#21788)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents in a `sandbox` environment run each task in a paid remote
sandbox that an environment lease tracks
> - When a run stops, the control plane must release that lease, and
terminating an agent waits until every lease of the agent is released
> - In a load test, a run stopped after its sandbox was ready but before
its runner started never released its lease, so the sandbox kept running
and the agent stayed in "terminating" forever
> - In the same test, terminating an agent whose failed run had a
scheduled same-run retry waited about 18 minutes, until the stale retry
time
> - This pull request makes lease release finish for every stop window,
records any release that must wait with a retry time, adds a hint-driven
safety net, and makes termination cancel a pending native retry at once
> - The benefit is that a stopped run never leaves a paid sandbox
running, and agent termination finishes in seconds without manual repair
## Linked Issues or Issue Description
**What happened?**
A load test with ten OpenCode agents in a `sandbox` environment stopped
its agents mid-run (pause, then terminate). Two defects appeared.
1. A run cancelled while it selected its provider pack (after the
sandbox was acquired and the workspace was prepared, before the runner
started) kept its environment lease `active` for more than 40 minutes.
The agent stayed in lifecycle `terminating`. The lifecycle worker
retried every pass and wrote `updatedAt` each time with no recorded
reason. Delete returned 409 "Complete termination before deleting the
agent".
2. A failed native run with a scheduled bootstrap retry kept its lease
until the stale retry time. Agent termination took about 18 minutes. The
run still showed "retry_scheduled" after the agent was terminated.
**Expected behavior**
A stopped run releases its sandbox lease within seconds. If release must
wait, the wait has a recorded reason and retry time, and a background
pass finishes it. Terminating an agent cancels a pending retry, releases
its lease, and reaches `terminated` promptly.
**Steps to reproduce**
1. Start a native run in a `sandbox` environment.
2. Stop it after the sandbox is acquired and the workspace is prepared,
before the runner starts.
3. Pause and then terminate the agent.
4. See that the lease stays `active` and the agent stays `terminating`.
**Paperclip version or commit**
Reproduced on `master` at
canary/v2026.1115.0-canary.18
|
||
|
|
b4f71fc276 |
perf(recovery): stop re-checking settled workspace flags and snapshot ledger rows every pass (#21795)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The server runs all background recovery on one Node main thread, and
each database statement costs time on it
> - The ownership reconciliation pass settles in-progress records every
30 s while it has records to watch
> - Two of its kinds listed records that can never change again: dirty
remote workspaces whose writer reached runtime, and snapshot ledger rows
that are uploaded or still held by a stale writer's epoch
> - Each such record cost a transaction, a row lock and up to four reads
on every pass, forever, and kept the 30 s watch armed on an idle
instance
> - This pull request lists only the records that the settle step can
still act on, locks the writer run only when the settle will write, and
skips one stranded-task read that can never match a native run
> - The benefit is that recovery work follows activity, not history
## Linked Issues or Issue Description
**Describe the problem**
On a hosted instance with 7 native runs, the ownership pass ran about
7.6 + 3.2 + 3.5 + 2.8 statements per second over records that cannot
change, and took a row lock on `heartbeat_runs` for each stale dirty
workspace on every pass. At 1,000 agents this becomes thousands of
statements per second.
**Steps to reproduce**
Seed about 100 dirty remote workspaces whose writer run reached runtime
and ended, 50 pending ledger rows still held by those writers, and 100
uploaded ledger rows. Run the ownership reconciliation worker. The new
census test
`server/src/__tests__/recovery-settled-records-census.test.ts` does this
on the embedded database.
**Expected behavior**
A settled record costs zero statements on later passes. An idle instance
runs no SQL from this pass.
**Actual behavior**
On master, one pass costs 714 statements and 150 row locks. An idle half
hour costs 40,800 statements and 9,000 row locks (the settle watch
re-checks every 30 s).
Refs #21765 (the stranded-assignment hint scoping that this rebases on)
## What Changed
- `remote_workspace_unsaved_files` lists a dirty workspace only when its
writer run can still be a preparation-only writer: stage `preparing`, no
`runtime_mode_resolved_at`, no process, session or runner id, no
finalization row and no work-started event (or no writer at all). The
SQL lives with the settle step in
`remote-workspace-unsaved-files-settlement.ts`.
- The unsaved-files settle reads the writer run without a lock. It takes
`for update` only after every check passes, just before it writes, and
checks the locked row again.
- `remote_snapshot_ledger` lists only `pending` rows with no object key,
whose operation is a workspace save, and whose workspace epoch has moved
past the row's writer. Uploaded rows belong to snapshot retention, as
before.
- `terminalRunSettlingUntil` in `recovery/service.ts` skips its
`heartbeat_runs` read when the latest run already in memory is not a
legacy run of another server boot. `getLatestIssueRun` now also selects
`runtime_mode` and `controller_boot_id`. The live-execution check still
runs first.
- New census test with a staging-shaped fixture, a statement counter
grouped by statement shape and caller, and prompt-pickup checks. New
red/green test for the stranded-task lease read.
Census (embedded PostgreSQL, same fixture, base
|
||
|
|
c2e7df2dcc |
fix(workspaces): use project storage for a fresh retry after unowned task storage (#20756)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A project can use a storage provider plugin. That plugin prepares each task's git worktree > - A board Retry after a failed run can start a fresh attempt. The fresh attempt starts from the project source > - `preparePluginWorkspace` returns `null` when the task is still bound to a workspace that no storage provider owns. It does this also for a fresh attempt > - On a plugin task environment, the host then reserves an empty adapter-managed "Fresh task workspace" with no repository. The runner fails in it. The next Retry finds that empty workspace bound to the task, returns `null` again and reserves another empty workspace. The task can never start again > - This pull request lets a fresh attempt use the project storage provider even when the previous workspace is not provider-owned > - The benefit is that one Retry gives the task a normal provider worktree again ## Linked Issues or Issue Description **What happened?** Tasks moved off an older execution environment kept a binding to a workspace that no storage provider owned. A board Retry selected `fresh_isolated_workspace`. `preparePluginWorkspace` returned `null` because the bound workspace was not plugin-owned. The host fell back to `reserveFreshPluginTaskWorkspace`, which made an empty `adapter_managed` workspace (strategy `project_primary`, no repository, no recovery snapshots). The run failed in about one second with "Plugin-provided Runner execution operation unavailable; reconcile the same task before retrying". Each later Retry made another empty workspace, chained to the last one. **Steps to reproduce** 1. Use a project with a storage provider plugin that supports fresh workspaces. 2. Bind a task to a workspace that the provider does not own (for example an adapter-managed workspace). 3. Call `preparePluginWorkspace` for a fresh attempt with that workspace as the previous workspace. **Expected behavior** The storage provider prepares a new worktree for the fresh attempt. **Actual behavior** `preparePluginWorkspace` returns `null` and never calls the provider. ## What Changed - `server/src/services/plugin-workspaces.ts`: the early `return null` for a previous workspace that no provider owns no longer applies to a fresh attempt. The fresh attempt then uses the project's storage provider, as for a task with no bound workspace. - `server/src/__tests__/plugin-fresh-workspaces.integration.test.ts`: new test "prepares provider storage for a fresh retry that follows a workspace the provider does not own". ## Verification - The first commit adds only the test. CI on that commit (`d09ad235`, [failed job](https://github.com/paperclipai/paperclip/actions/runs/42976154239/job/130393285731)) shows only this test fail on the current code (`preparePluginWorkspace` returns `null`). - The second commit adds the fix. CI shows the test pass. - The other tests in the file keep the fresh-attempt guarantees: an older provider still fails closed, a changed binding revision still stops the call, and a lost reply still reconciles to the same operation. - I did not run the suite locally. The test needs embedded Postgres, and this environment does not install packages. CI runs it. ## Risks - Low to medium risk. A fresh attempt in a project with a storage provider now uses that provider even when the previous workspace was local or adapter-managed. Before, such an attempt used the fallback path. This matches the project's storage policy, and a fresh attempt does not reuse the previous workspace's files. - Projects without a storage provider are not affected: the function still returns `null` for them. - Non-fresh preparation is not affected. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), through Claude Code with tool use (shell, file edits). Standard reasoning mode. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: lucktastic <13757243+lucktastic@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
c6cdd57f7a |
perf(native-runtime): reuse a same-content image provider pack across builds (#21784)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A remote agent run needs the provider pack (Node, OpenCode and the Runner bridge) inside its sandbox before the provider can start > - The server reuses a pack that is installed in the sandbox image only when its full manifest equals the server's own manifest > - The manifest includes `runnerSourceRevision`, which is the commit that built the server, so an image pack fails this check on every later server build, also when every file in the pack is the same > - The run then uploads the whole pack again, which cost about 15 s per task in a recent load test > - This pull request accepts an image pack whose sealed content equals the expected pack and that differs only in build provenance, and caches verified archive hashes in the server process > - The benefit is that a sandbox image stays usable across server releases that do not change the pack, and the server does less hashing on its main thread ## Linked Issues or Issue Description Refs #21781 (merged), which names the failed check on the run spans. **What existing behavior does this improve?** Selection of the remote provider pack for a new sandbox (`provider_pack.select`). **Subsystem affected** Server: native runtime, remote provider pack verification and upload. **Current behavior** `provider_pack.verify_preinstalled` compares the complete manifest of the image pack with the server's manifest. Any difference fails, including `runnerSourceRevision` and `runnerInputsDigest`, which only record where the pack was built. The run then uploads the full pack. The server also hashes the same archive or pack directory again for every task. **Proposed behavior** When the run is not bound to a saved pack and the operator did not select a custom pack, the remote verification accepts an image pack when (1) its own digest seals its payload, and (2) its schema and payload equal the expected pack after the two provenance fields are removed. Every artifact, the `dist` tree, the candidate trees, the package versions, the Node version, the target platform and the OpenCode version are still checked in the sandbox, against the same values. The server checks the reported manifest again and selects it through the pack authority, as the compatibility path already does. An upload still happens for different content, a corrupted or partial pack, or no pack. The server also remembers a verified transport archive (by path, inode, size and times), so it does not hash it again for the next task. **Reason and benefit** In the harness (fake sandbox, 150 ms per command, 300 ms + 10 MiB/s per transfer, 24 MiB pack), the cross-build case goes from 4.1 s and 24 MiB uploaded to 1.1 s and 0 bytes. Other cases do not change. **Breaking changes** None. A saved or reattached session, and an operator-selected custom pack, still require the exact pack identity. ## What Changed - `remote-provider-pack-verification.ts`: new `acceptProvenance` option. The remote script accepts a sealed pack with the same content and prints its manifest. The server checks that manifest (`sameContentManifest`) and returns it. A forged or different report fails with `installed_manifest_invalid`. - `native-session-executor.ts` (`prepareRemoteProviderPackOnTarget`): staged and preinstalled verification use the option when `acceptProvenanceDifferences` is set. The function returns the selected manifest. The run caller sets the option only through `providerPackProvenanceReuseAllowed` (not retained, no explicit custom pack, selectable authority), and selects a different manifest through the pack authority. - `provider-pack-layers.ts`: `verifyArchive` skips the hash for an archive it already verified with the same file identity. - `remote-provider-pack.ts`: `providerPackProvenanceReuseAllowed` gate, with tests for each case. - `provider-pack-selection-harness.test.ts`: new harness. Uploads go through the workspace confinement check of the Daytona sync guard. It runs the real remote verification in local processes, with modelled sandbox latency and transfer speed, for five cases, before and after. ## Verification - `cd server && npx vitest run src/services/native-runtime/provider-pack-selection-harness.test.ts src/services/native-runtime/provider-pack-archive.test.ts src/services/native-runtime/remote-provider-pack-verification.test.ts src/services/native-runtime/remote-provider-pack.test.ts src/services/native-runtime/retained-provider-pack-source.test.ts src/__tests__/kimi-connection-probe.test.ts` passes. - `npx vitest run src/services/native-runtime/native-session-executor.test.ts src/services/native-runtime/native-artifact-preparation.test.ts src/services/native-runtime/remote-provider-pack-authority.test.ts` passes (1541 passed, 7 skipped). - Red/green: with the source changes reverted, the harness case `same_content_other_build` and the archive cache test fail. With the changes, they pass. - Harness output (load average about 9 on a shared machine; modelled time = wall time + modelled sandbox latency): | Case | Before | After | | --- | --- | --- | | Exact image pack | preinstalled, 1.1 s, 0 MiB | preinstalled, 1.0 s, 0 MiB | | Same content, other build | uploaded, 4.1 s, 24 MiB | preinstalled, 1.1 s, 0 MiB | | Different content | uploaded, 4.1 s, 24 MiB | uploaded, 4.0 s, 24 MiB | | Corrupted image pack | uploaded, 4.0 s, 24 MiB | uploaded, 4.1 s, 24 MiB | | No image pack | uploaded, 3.7 s, 24 MiB | uploaded, 3.8 s, 24 MiB | - The pack step runs no database statements. - `npx tsc --noEmit -p server/tsconfig.json` passes. ## Risks - This changes which provider pack the server accepts. It accepts a different manifest digest only when all content fields are equal and every file is hashed in the sandbox, and only for a run that is not bound to a saved pack. The selected manifest goes through the existing pack authority, so launch profiles and receipts bind the installed pack's own identity. - The archive hash cache trusts each archive file's identity (inode, size, times) on the server. The sandbox still checks the uploaded bundle digest and verifies every uploaded file, so a changed local file still fails closed. - This does not help when the image pack has other content (for example a different contract digest). See the hand-off notes for the image build. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>canary/v2026.1115.0-canary.17 |
||
|
|
b5447432bd |
fix(ui): keep the active company on Cases and Status navigation links (#21794)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The board UI builds company-prefixed links, such as `/PAP/cases`,
from short paths in the sidebar
> - The Cases and Status links use `/cases` and `/status`, but these
roots were not in the board route list
> - The router read `cases` as a company prefix and left the link
unprefixed, so the link only worked through the unprefixed redirect
> - When a person clicked Cases before the company list loaded, the
redirect picked an arbitrary company and showed its empty case list
> - The same race made `tests/e2e/cases-read-only.spec.ts` fail in "ci /
e2e shard (2/8)" on unrelated PRs (#21683, #21715, #21765)
> - This pull request adds both roots to the board route list and makes
the e2e test force the race
> - The benefit is that Cases and Status always open in the company the
person is in, and the flaky e2e test becomes deterministic
## Linked Issues or Issue Description
Refs #21683
Refs #21715
Refs #21765
**What happened?**
The read-only Cases e2e test failed about once per day with `Expected:
3, Received: 0` links in the case list (job 132875164249 on #21765, job
132680942843 on #21715). The failure screenshot shows the sidebar of a
different company ("Passive artifacts …", created by the previous spec)
and "No cases yet". The server log shows that the page asked for the
cases of the other company and never for the test company.
**Expected behavior**
A click on the Cases or Status link opens that page for the company in
the URL, also when the click happens before the company list has loaded.
**Steps to reproduce**
1. Have two or more companies.
2. Open `/<PREFIX>/issues` on a slow connection (the e2e test holds `GET
/api/companies`).
3. Before the company list loads, open More and click Cases.
4. The URL becomes `/cases`. When the list loads, the app redirects to
`/<OTHER>/cases` and shows another company's empty case list.
**Paperclip version or commit**
`master` at
canary/v2026.1115.0-canary.16
|
||
|
|
df9914dbd9 |
fix(native): let the outgoing server checkpoint its idle warm session while the incoming follow-up waits for it (#21791)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A deploy swaps servers with an overlap. A warm agent session stays with the outgoing server that ran its last turn. > - Since #21774, a follow-up that the incoming server claims during the overlap waits (`native_controller_recovery_waiting_for_owner`) and checks again every 5 s. > - About 1 s after the traffic switch (#21627), the outgoing server tries to save and close its idle warm session. The checkpoint fence refuses that save with `native_runner_checkpoint_owner_lost`. The task's current run is the parked follow-up. That run has its own finalization row under another controller boot, and the fence does not accept that kind of successor. > - The close retry from #21782 fails the same way each time. So the follow-up waits until the outgoing server exits, or until the predecessor's finish time plus the idle window plus 2 min. In the CHA-5 case that was about 225 s, and the worst case is about 425 s. Speed is a requirement. > - This pull request records which owner a parked follow-up waits on, and lets the fence accept the outgoing owner's final save when the follow-up is parked on exactly that owner and has not started. > - The benefit is that the outgoing server saves and closes the session on its first attempt, and the follow-up continues at its next 5 s check. ## Linked Issues or Issue Description Refs #21774, #21782, #21697, #21627, #21279 **What happened?** During a release-swap overlap, a warm follow-up was claimed by the incoming server while the outgoing server still owned the idle warm session. The follow-up waited correctly. The outgoing server's idle close failed with `native_runner_checkpoint_owner_lost` on every attempt. The wait ended only when the outgoing server exited, about 225 s after the follow-up was created. **Expected behavior** The outgoing server saves and closes the session while the follow-up waits. The follow-up sees the saved session and continues within one 5 s check. **Steps to reproduce** 1. Run a warm native session on a remote sandbox and let a run finish on server A. 2. Start server B with an overlap and switch traffic to it. 3. Post a follow-up comment so that B claims the follow-up while A is still alive. 4. A's idle close fails with `native_runner_checkpoint_owner_lost`, and the follow-up waits until A exits. **Paperclip version or commit** master `d14a3b88b1`. ## What Changed - `NativeControllerRecoveryWaitingError` carries the owner that the follow-up waits on: the predecessor run and its controller boot. `waitForLiveWarmSessionOwner` and the live-owner wait in `inspectRemoteWarmSessionRecovery` set it. - `deferNativeControllerRecovery` stores that owner in the finalization `failureDetail.waitingForOwner`. The executor passes it through. - `validateNativeRunnerCheckpointOwner` accepts one more successor kind: a parked follow-up waiting on this exact owner. All of these must be true, or the fence still refuses: - The checkpoint holds an accepted result and has no new work (unchanged rule). - The follow-up finalization is `retryable_failure` with `native_controller_recovery_waiting_for_owner`, `attempt = 0`, no result, and no lease owner (no claim is running it). - `waitingForOwner` names this owner's run and controller boot. - The follow-up run is a running native run for the same company and task, on the same native session as the owner's run. - The follow-up has no process fields, no own process-start events, and no checkpoint rows. - The fence locks the follow-up's finalization row (`for share`), so a resumption claim and the save cannot interleave. - Tests in `native-controller-recovery.integration.test.ts` (real Postgres): - The outgoing server saves and closes its idle warm session while the incoming follow-up waits, and the follow-up resumes within one recheck. - Negative cases: the save is still refused when the follow-up has its own process, its own process-start event, its own checkpoint, or a live resumption claim, or when it waits on a different owner boot or run. ## Verification - Red on master: with the `native-runner-checkpoint.ts` change removed, the scenario test fails with `native_runner_checkpoint_owner_lost` at the outgoing server's first close attempt. The follow-up keeps waiting. Green with the change. - The negative cases pass with and without the change. - Measured in the test scenario (warm idle window 600 s): - Before: the follow-up waits until the outgoing server exits, or about 720 s (finish + 600 s idle window + 120 s margin). - After: the outgoing save and close succeed on the first attempt. The follow-up is claimable and its wait resolves at `retryAt`, 4998 ms after it parked. - `vitest run` of `native-controller-recovery.integration.test.ts`, `native-runner-restart-recovery.integration.test.ts` and `native-harness-backup-storage.integration.test.ts`: 349 passed, 14 skipped. - `vitest run` of `native-session-executor.test.ts`, `remote-warm-session-recovery.test.ts`, `native-runner-checkpoint-save.test.ts`, `native-runner-checkpoint-retirement.test.ts` and `native-runner-checkpoint-authority.integration.test.ts`: 1566 passed, 7 skipped. - `tsc --noEmit` for `server` passes. - `pnpm verify` exits 2 (opaque setup dependency). Required CI supplies the broad coverage. - Not run: a release-swap wave. The fix is not yet measured on a live overlap. ## Risks - The fence is wider. It accepts only a final save with an accepted result and no new work, and only while the current run is parked on this exact owner and has not started. A follow-up that has a claim, a process, process-start events or checkpoint rows still causes a refusal. - A follow-up that a server parked before this change has no `waitingForOwner`. The fence refuses its owner's save as before. At its next recheck, the follow-up parks again with the owner recorded. - While a resumption claim runs the follow-up's recheck (a short window every 5 s), the save is refused. The close retry from #21782 then tries again. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
cec8c2b3f4 |
perf(recovery): name the run or task in run writers' recovery hints instead of the company (#21765)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The server runs issue recovery queues on one Node main thread. The `stranded-assignments` and `assigned-work-invariant` queues check assigned tasks that have no live run. One stranded check costs about 19 database statements. > - #21552 scoped run hints to the run's task graph. #21729 made the 5-minute backstop re-check only changed tasks. A hint with only a company id still makes the queue re-check every open task of the company, 25 per pass. > - Under load on staging (7 native runs), the stranded check ran about 6.7 times per second, and its statement bundle was more than a third of all statements. An activity fixture shows the cause: ordinary run writers send a company-only hint once per run, and one company hint costs about 850 checks. At 1,000 agents this work never stops. > - This pull request makes those writers name the run (or the task) they already know. A run that has no task now resolves to no tasks instead of the whole company. The changed-tasks backstop no longer counts an agent's own run heartbeat as a change to every task of that agent. > - The benefit: with 7 runs a minute plus one taskless run, stranded checks drop from 18.4 to 1.0 per second and stranded-check statements from 344 to 7 per second. Real company-wide changes (pause lifted, budget, company status, tree holds) keep the full company pass. ## Linked Issues or Issue Description Refs #21729, #21552, #21676, #21724 **What existing behavior does this improve?** The issue recovery task queues (`stranded-assignments`, `assigned-work-invariant`, `resolved-dependencies`, `stale-issue-locks`) and the writers that wake them. **Subsystem affected** Server: issue recovery background work and the run lifecycle writers that publish recovery hints. **Current behavior** Many run lifecycle writers know the run but publish a company-only hint, for example `notifyExecutionRunWork(tx, { companyId })` in the native finalizer and projection, `settleNativeTerminalWake`, the claimed-wake settlement, and an environment lease release of a run without a task. A queued run without a task publishes a company hint. A graph hint for a run without a task resolves to "whole company". A comment sends a company hint to `assigned-work-invariant` and `resolved-dependencies`. Each of these re-checks all open tasks of the company. Also, the changed-tasks backstop treats any change to the assignee agent's row as a change to all of that agent's tasks, and every run start and finish writes that row. **Proposed behavior** A writer that knows the run sends the run's graph hint. A writer that knows the task sends the task hint. A run without a task affects no task queue entry. An agent row change that is only the agent's own run heartbeat does not mark its other tasks changed. Company-wide facts keep the company hint. **Reason and benefit** The work is now proportional to activity, not to (activity x history). See Verification. ## What Changed - `server/src/services/issue-recovery-notifications.ts`: - `notifyIssueRecovery` with a task sends the task hint to all four task queues (before: only to `stranded-assignments` and `stale-issue-locks`; `assigned-work-invariant` and `resolved-dependencies` got the company hint). Only comments (`issue-comment-writes.ts`) call it with a task. - `resolveIssueGraph` returns `[]` when a run-only graph has no task. A missing task seed still returns `null` (whole company). - `server/src/services/recovery/work.ts`: a load for an empty graph returns no rows. The `assigned-work-invariant` queue emits the same `check` debug event as `stranded-assignments` (with `PAPERCLIP_DEBUG_WORK_SIGNALS=assigned-work-invariant`), with the load reason. - `server/src/services/recovery/changed-tasks.ts`: the assignee-agent part of the changed-tasks condition skips an agent whose last write was its run heartbeat (`last_heartbeat_at = updated_at`). A resume, pause or configuration write (no heartbeat stamp) still counts. - `server/src/services/heartbeat/queue.ts`: a queued run without a task gets its id before the insert and publishes a run hint, not a company hint. The daily-cap comment release publishes the run's graph hint. - `server/src/services/environments.ts`, `work-intent-outbox.ts`, `chat-inbound-wake-notice-notifications.ts`: a lease release passes its run id. `promoteDeferredWorkIntents` scopes `lock_released` with only a run to that run's graph. `notifyChatInboundWakeNotices` accepts an optional `runId` for its settlement hint. - Native and recovery run writers now pass the run id they already have to `notifyExecutionRunWork`, `writeExecutionControl` or `notifyExternalConversationSettlement`: `native-run-finalizer.ts` (3), `native-run-projection.ts` (2), `native-session-executor.ts` (3), `wake-request-claimed-settlement.ts`, `native-superseded-retry.ts`, `native-safe-replacement.ts`, `native-restart-recovery.ts` (2), `native-failed-workspace-recovery.ts`, `native-recovery-preparation-failure.ts`, `execution-control-reconciliation.ts`, `execution-recovery-resolution.ts` (2). These are the native equivalents of the legacy run paths in `heartbeat/queue.ts`, which already pass the run id. - Tests: - `server/src/__tests__/issue-recovery-activity-hints.test.ts` (new): staging-shaped company (780 in-review tasks with succeeded native runs, 40 in progress, 30 to-do, 50 agents) on embedded PostgreSQL, the real recovery, dispatch and retry-cancellation workers, a fake clock, and real process-adapter runs. Each run is woken by a user comment through `heartbeat.wakeup`, starts, calls back to post an agent comment and move the task to `in_review`, and finishes. One (or five) taskless runs a minute run beside them. It counts checks by queue and reason, task-queue signals by caller, and statements by owner. It asserts the workload (runs started and succeeded, agent comments, run events, zero agent write failures, at most 10% failed runs), then no company check and no company-only task-queue signal. In CI it runs the 7-runs-a-minute workload for 4 minutes (about 1 minute of real time). With `RECOVERY_ACTIVITY_REPORT=<file>` it runs both workloads at full length, attributes statements by stack, and writes the numbers below to that file. - `server/src/__tests__/issue-recovery-deadlines.test.ts`: pickup tests for each narrowed writer (see Verification). ### Publishers of company-only (or `*`) hints to the recovery task queues "Per" is how often the writer fires on a busy instance. "Known" says whether the run or task is known at the call site. Narrowed in this pull request: | Call site | Event | Per | Known | Now | | --- | --- | --- | --- | --- | | `heartbeat/queue.ts:5396` `enqueueWakeup` | queue a run without a task (timer, manual, chat without a task) | each taskless run | run (id generated) | run graph (no tasks) | | `environments.ts:1780` `releaseLease` -> `notifyChatInboundWakeNotices` | environment lease released | each run without a task | run | run graph | | `environments.ts:1792` `releaseLease` -> `promoteDeferredWorkIntents(lock_released)` | lease released | each run without a task | run | run graph | | `environments.ts:1934` `releaseLeasesForRun` | leases of a run released | each run without a task | run | run graph | | `issue-comment-writes.ts:12` (`assigned-work-invariant`, `resolved-dependencies` only) | comment | each comment | task | task | | `native-runtime/native-run-projection.ts:54` `settleNativeTerminalWake` | native run's wake claim settled | each native run finish | run | run graph | | `native-runtime/native-run-projection.ts:177` | native run status projected | each native run finish | run | run graph | | `native-runtime/native-run-finalizer.ts:408`, `:629`, `:2080` | native finalization terminal / failure / projection | each native run finish | run | run graph | | `native-runtime/native-session-executor.ts:13018`, `:13085`, `:13284` | native execution control deadline, cancelled retry, pre-result settlement | each native run end | run | run graph | | `recovery/wake-request-claimed-settlement.ts:67` | claimed wake settled after the run | each run whose claim is reconciled | run | run graph | | `heartbeat/queue.ts:590` | daily-cap comment key release | each run cancelled at the daily cap | run | run graph | | `native-runtime/native-superseded-retry.ts:42`, `native-safe-replacement.ts:613`, `native-restart-recovery.ts:1033`, `:1935`, `native-failed-workspace-recovery.ts:164`, `native-recovery-preparation-failure.ts:30`, `execution-control-reconciliation.ts:270`, `execution-recovery-resolution.ts:830`, `:965` | native recovery terminal writes | each recovered run | run | run graph | | `resolveIssueGraph` for any run-only hint of a run without a task (for example `setWakeupStatus`, `run-lifecycle.ts:1570`) | run lifecycle | each taskless run event | run | no tasks | Left with the company hint (company-wide fact, or the graph is not known): | Call site | Event | Per | Why kept | | --- | --- | --- | --- | | `budgets.ts:371`, `:386`, `:429`, `:437`, `:454` | budget policy changed, scope paused or resumed | operator or budget event | eligibility of every task can change | | `work-intent-outbox.ts:79` with `pause_lifted`, `tree_hold_released`, `scheduling_resumed`, `reconciler_backstop`, and `accounting_settled` with an active hard stop | hold release | operator event; `accounting_settled` once per run only with a hard-stop policy | can admit any task; guarded by "keeps the company pass for hold releases that can admit any task" | | `companies.ts:77`, `:350`, `:353`, `:361`, `:512`, `:522` | company created, paused, resumed, archived | operator | company-wide | | `agent-lifecycle.ts:65`, `:72`, `agent-lifecycle-driver.ts:29`, `agent-configuration.ts:130`, `:159`, `agent-deletion.ts:61` | agent paused, resumed, configured, deleted | operator | can change many tasks, and other agents' review paths | | `access.ts:570`, `:752`, `:912`, `:926`, `:960`, `:975`, `:1175`, `board-claim.ts:129`, `:142`, `middleware/auth.ts:706` | membership and authority changes | operator, first sign-in | company authority | | `chat-channels/endpoints.ts:461`, `identity-links.ts:323`, `resource-reconciliation.ts:290`, `slack/setup-services.ts:50`, `chat-github-management.ts:182`, `:352`, `chat-work-notifications.ts:50` | chat authority changes | operator | company authority | | `instance-settings.ts:666`, `:669`, `:680` (`*`) | instance setting changed | operator | instance-wide | | `issue-tree-control.ts:809`, `:889`, `:1236`, `:1339` | tree hold created or released | operator | many tasks | | `company-deletion.ts:83`, `company-portability.ts:6560` | company delete, import | rare | many tasks | | `issues.ts:8015` (blockers replaced), `:11165` (task created), `:12685`, `:12699` (task deleted), `:12991`, `:14618` | task graph changes | per task create or blocker edit | graph moves; #21552 kept these | | `documents.ts:615`, `:673`, `:757`, `:808`, `:860` | document delete and restore | rare | #21552 kept these | | `task-watchdogs.ts:854`, `:1841` | watchdog changes | rare | #21552 kept these | | `system-recovery.ts:17`, `:70`, `task-retry.ts:294`, `workspace-recovery-capture-policy.ts:59`, `workspace-recovery-stop-proof.ts:122` | system and workspace recovery | per recovery | not shown to stay in one graph | | `agent-conversations.ts:607`, `chat-channels/telegram/attachment-failure.ts:83`, `native-runtime/chat-attachment-reuse.ts:1754`, `native-runner-file-handoff.ts:1317` | conversation and attachment paths | per failure | not shown to stay in one graph | | `wake-queue/adapters/postgres.ts:376`, `:410`, `:555`, `:1169`, `routes/issues.ts:14552`, `issues.ts:12070`, `:12171`, `heartbeat/run-control.ts:1128`, `:1770`, `projects.ts:100`, `:1051`, `environments.ts:1617` (`notifyChatInboundWakeNotices` without a task) | saved notice and wake changes | per manual wake or project change | company admission state; not reached by run activity in the fixture | | `heartbeat/run-state.ts:1746`, `native-harness-backup-expiration.ts:356`, `:526`, `explicit-native-continuation.ts:705` | session goal clear, backup expiry, explicit continuation | rare | no run id at the call site | | `fast-responses.ts:201`, `:559`, `:672`, `:1053`, `chat-github-response-comments.ts:62`, `:229`, `:440`, `chat-github-reviews.ts:1063` (`notifyChatResponseAvailability` with a delivery only) | chat response availability | per chat response | no task at the call site; not measured here (chat paths are not in the fixture) | Per-run-event path: appending run events, saving native checkpoints, renewing leases and streaming deltas publish no recovery task-queue hint. `heartbeat-run-events.ts` reaches only live-run recovery queues (`native.process_*`, `turn.*` events), chat progress, response presentation, and `notifyNativeStatusReassessment` for control-plane evidence events only. Checkpoints and lease renewals reach `notifyLiveRunRecovery`, which does not include the task queues. A run's start and finish publish its graph hint from several writers (about 20 task-queue signals per legacy run in the fixture, for example `notifyChatRunMilestones` in `run-identity.ts` activation, `claimQueuedRun`, `run-completion.ts`, `setWakeupStatus`). They coalesce into about 3 checks of the run's own task per run. That is not one hint, but it is bounded by the run's graph. This pull request does not change it. ## Verification - Activity fixture (`issue-recovery-activity-hints.test.ts`), checks per second by queue and reason, and statements per second. Each workload minute is exactly 60 s of fake-clock time, and the rates use the elapsed fake-clock time. "Before" is this branch's test against the `origin/master` service files. Both versions did the same work (7 runs/min: 150 runs started, 143 to 145 succeeded, 133 to 135 agent comments, about 1,280 run events; 50 runs/min: 629 to 630 started, 620 to 622 succeeded, about 590 agent comments, about 5,550 run events). The test now asserts this workload and zero agent write failures before it accepts the counts. | | master | this branch | | --- | --- | --- | | 7 runs/min + 1 taskless run/min (600 s), stranded checks/s | 18.30 company + 0.07 hint | 0.91 hint + 0.06 backstop, 0 company | | 7 runs/min, statements/s in stranded checks | 343.9 | 7.2 | | 7 runs/min, all statements/s | 577.5 | 220.4 | | 7 runs/min, company-only task-queue signals | 710 | 0 | | 50 runs/min + 5 taskless runs/min (360 s), stranded checks/s | 16.86 company + 0.31 hint | 1.05 hint, 0 company | | 50 runs/min, statements/s in stranded checks | 314.0 | 7.4 | | 50 runs/min, all statements/s | 1,717.6 | 1,387.5 | | 50 runs/min, company-only task-queue signals | 2,864 | 0 | "Statements in stranded checks" counts statements whose stack includes `reconcileStrandedAssignedIssues`. The other statements are the runs themselves. Recovery load queries (graph lookups and hint loads) are 12.5 to 17.2/s at 7 runs and 69 to 72/s at 50 runs in both versions. `stale-issue-locks` saved deadlines check about 1 to 1.6 tasks/s in both versions. - Without the taskless runs, master already showed the backstop degrade under activity: with 7 runs a minute, 2.46 backstop checks/s, because every run writes its agent's row and marks all of that agent's tasks changed. With this branch: 0.27/s. - New tests in `issue-recovery-deadlines.test.ts`, red on `origin/master` service files and green here: - "re-checks a run's own tasks for its settlement, lease and dispatch writers, and nothing for a run without a task": for `notifyRunDispatch`, `notifyExecutionRunWork`, `notifyExternalConversationSettlement`, `notifyChatInboundWakeNotices` and `promoteDeferredWorkIntents(lock_released)` with a run id, the run's graph (task, parent, dependent) is checked within one pass and none of 300 backlog tasks is. For a run without a task, no task is checked. A comment checks only its task in `stranded-assignments` and `assigned-work-invariant`. - "counts an agent's own run heartbeat as no change to its other tasks, and a resume as a change". - "resolves a run without a task to no tasks and a missing task to the whole company". - Unchanged and passing: the #21729 pickup tests in `issue-recovery-idle-backstop.test.ts` (finished run, user comment, raised budget, resumed agent within a minute; missed hint within the backstop bound; outside change within the hourly pass), the #21552 tests in `issue-recovery-deadlines.test.ts` (including "keeps the company pass for hold releases that can admit any task" and "falls back to the company pass when a run affects more tasks than a hint names"). - Regression, embedded PostgreSQL: `issue-recovery-deadlines`, `issue-recovery-work-debug-attribution`, `issue-recovery-signal-fanout`, `issue-recovery-work-lifecycle` (42 tests); `native-run-projection-work`, `native-recovery-notice-work`, `muse-terminal-settlement`, `heartbeat-held-comment-wake`, `superseded-workspace-export-recovery`, `native-recovery-preparation-failure`, `upgrade-readiness-post-result`, `chat-wake-conversation-batch`, `environment-service`, `retained-native-session-cleanup-work`, `slack-conversation-lifecycle`, `workspace-recovery-dependency-finalize`, `native-failed-run-owner-release`, `chat-run-milestone-work` (462 tests). - `tsc --noEmit -p server/tsconfig.json` passes. ## Risks - **Recovery time bounds. The owner must see this.** No hint-driven pickup becomes slower for a task in the run's graph. Two missed-hint bounds change: - An agent change that sends no hint and is followed by a run heartbeat of the same agent inside the 5-minute window (for example a resume whose hint was lost, after which the agent ran another task) is now found by the hourly full pass, not the 5-minute changed-tasks pass. A normal resume sends `pause_lifted` and is checked at once (tested). - A task outside the run's graph whose recovery depended on the end of a taskless run, or on the native finish writers above, is now found by its own hint, the 5-minute changed-tasks pass (its own rows), or the hourly full pass. Before, any run finish re-checked the whole company. I found no recovery decision that waits on another run of the agent: concurrency waits are queued runs that dispatch resumes. - A run-only hint for a run row that does not exist resolves to no tasks (it was the whole company). A run row is not deleted while its tasks are open, except by company deletion, which sends the company hint. - Native finish writers now send run-scoped hints to the live-run recovery queues too (`notifyLiveRunRecovery`, `notifyLiveResourceRecovery`, attention retention). The legacy run paths already send these with the run id. - The fixture uses process-adapter runs, not native runs. The native writers are narrowed by code reading. Their effect on the hosted staging instance is not verified. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
10ddb26dbb |
test(server): accept dash's exit code for a sandbox-denied shell write (#21792)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Runner workspace bridge runs agent commands in a `bwrap` sandbox that binds only `/usr`, `/bin`, `/lib` and the task workspace > - A server test checks that a sandboxed command cannot read or write a file outside the workspace > - The test only runs where `/usr/bin/bwrap` exists. CI jobs that the gate sends to GitHub's `ubuntu-latest` now get the `ubuntu-26.04` image, which has `bwrap`, so the test runs on CI for the first time > - There it fails on every run, because it expects the write attempt to exit with 1, but Ubuntu's `/bin/sh` (dash) exits with 2 when a redirection fails > - This pull request checks that the write fails and that the file is unchanged, without depending on the shell's exit code > - The benefit is that the "General tests (server (7/12))" job passes again on the new image, and the test still proves the sandbox denial ## Linked Issues or Issue Description **What happened?** `server/src/services/native-runtime/runner-bridge-tools.test.ts` › `Runner workspace bridge > runs useful commands but denies files outside the workspace and injected credentials` fails in "ci / General tests (server (7/12))" on unrelated branches, on the first run and the retry: - PR #21765, head `98197c87e9` - branch `feat/run-loop-detector`, run 43732047843, head `b8c6e27429`, job 132866487949, image `ubuntu-26.04` version 20261111.196.1 ``` AssertionError: expected { exitCode: 2, signal: null, …(3) } to match object { exitCode: 1 } ❯ src/services/native-runtime/runner-bridge-tools.test.ts:180:19 ``` On jobs that ran on the RunsOn `ubuntu24` image there is no `/usr/bin/bwrap`, so the test is skipped (7 skipped instead of 6). **Expected behavior** The test passes wherever the command sandbox is available, and it fails if the sandboxed command can read or change the outside file. **Steps to reproduce** 1. In a Debian or Ubuntu container, run `/bin/sh -c 'printf stolen > "$1"' sh /nonexistent/dir/private.txt`. 2. dash prints `cannot create /nonexistent/dir/private.txt: Directory nonexistent` and exits with 2. bash exits with 1. `cat` of the same path exits with 1. 3. In the sandbox, the outside temporary directory is not bound, so the path does not exist. This is the same case. **Root cause** A stale test expectation, exposed by a CI image change. No commit on `master` changed the bridge, the sandbox or this test today (the last change to the test is from 2026-11-08). The test expected the shell write to exit with 1. That is bash behavior. On Ubuntu, `/bin/sh` is dash, which exits with 2 for a failed redirection. Until today no CI image had `bwrap`, so the test never ran on CI. **Production impact** None. The sandbox denies the access as intended: the `cat` step on the same run exits with 1 and its output does not contain the secret, and the write fails with `Directory nonexistent` because the outside directory is not visible in the sandbox. This is not a sandbox regression. ## What Changed - The write step now checks that the outside file still contains `PRIVATE` first, then that the command was not stopped and exited with a non-zero code. It no longer requires exit code 1 from the shell. - The read step and the success step are unchanged. ## Verification - Linux container (`node:24-bookworm-slim`, `/bin/sh -> dash`): the write command exits with 2 under dash and 1 under bash, and both are failures, as the test now accepts. - `npx vitest run src/services/native-runtime/runner-bridge-tools.test.ts` in `server/` on macOS: 11 passed, 6 skipped (the sandbox tests need Linux with `bwrap`). - I cannot run the sandbox test on this Mac. It runs in this pull request's CI only when the job lands on an `ubuntu-26.04` runner. ## Risks - Low risk. Only one test assertion changes. The test still fails if the write succeeds, because the file content check runs first and the exit code must be non-zero. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edit, GitHub CLI, Docker). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
3ee6049f71 |
perf(native-runtime): name why a preinstalled provider pack was not reused (#21781)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A remote agent run needs the provider pack (Node, OpenCode and the Runner bridge) inside its sandbox before the provider can start > - The server reuses a pack that is already installed in the sandbox image, or it uploads its own pack > - In a recent load test every task uploaded the pack, and this step was about 43% of the time to first token > - The compatibility check and the preinstalled-pack verification failed, but the run records did not say which check failed > - This pull request records the failed check on the run's performance spans > - The benefit is that the next run names the reason, so the image or the check can be fixed without guesswork ## Linked Issues or Issue Description **What existing behavior does this improve?** The run performance spans for provider pack selection. Today a failed `provider_pack.compatibility` or `provider_pack.verify_preinstalled` span only says `outcome: failed`. The code catches the error and drops it. **Subsystem affected** Server: native runtime, remote provider pack selection (`server/src/services/native-runtime`). **Current behavior** `provider_pack.select` records `reason: cached_packs_unverified` when it uploads. The compatibility check throws one generic message for about ten different conditions, and the caller swallows it. The remote verification script throws a generic `manifest mismatch` without the fields that differ. **Proposed behavior** Each failed check has a short, fixed name (for example `installation_unprotected`, `contract_digest`, `artifact:nodeCommand`, `manifest_mismatch:runnerSourceRevision`). The trace records this name as the `check` attribute on the failed span. `provider_pack.select` records the last preinstalled rejection as `check` when it uploads. The compatibility span records `target: image` or `target: workspace`. **Reason and benefit** Upload of the provider pack costs about 15 s per task on the test instance. To remove it, we must know which check rejects the image pack. The attribute holds only fixed check names and manifest field names, never paths or values. **Breaking changes** None. Error messages keep their existing text. The upload decision does not change. ## What Changed - `remote-provider-pack-verification.ts`: the compatibility check names the failed condition (`failureCheck`). The remote verification script prints a fixed `paperclip-pack-check=<name>` line, and for a manifest mismatch it names the differing manifest fields. Missing files now give a named check instead of a raw stack trace. - `native-run-trace.ts`: a failed `measure` records the error's `failureCheck` as the safe `check` attribute. - `remote-provider-pack.ts`: `provider_pack.select` records the last preinstalled rejection (or `no_preinstalled_pack`) when it uploads. - `native-session-executor.ts`: the `provider_pack.compatibility` span records `target`, the preinstalled-pack selection passes its rejection callback, and a failed link has the `link_failed` check. No other lines change. ## Verification - `cd server && npx vitest run src/services/native-runtime/native-run-trace.test.ts src/services/native-runtime/remote-provider-pack.test.ts src/services/native-runtime/remote-provider-pack-verification.test.ts src/__tests__/redaction.test.ts` passes (131 tests). - Red/green: with the source changes reverted and the new tests kept, 12 tests fail. With the changes, all pass. - The verification test runs the real remote script in a Node process: a provenance-only difference gives `manifest_mismatch:runnerSourceRevision`, a wrong artifact gives `artifact_digest:nodeCommand`, and a missing manifest gives `manifest_unreadable`. - `npx tsc --noEmit -p server/tsconfig.json` passes. ## Risks - Low risk. The change adds diagnostics only. The selection order, the checks themselves and the upload path do not change. - The `check` attribute is filtered by the existing safe-attribute list and a strict character pattern, so it cannot carry paths or secrets. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
46e5e4450f |
fix(runner): leave an undispatched tool call to the next controller on restart detach (#21786)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Paperclip Runner keeps an agent session alive while its controller restarts, for example during a deploy, and the next controller adopts it > - A Paperclip tool call can be between its committed input and its handler when the controller detaches > - In that window the old controller journaled "The turn stopped before this operation was dispatched" as the final, non-retryable result, but the turn did not stop > - The agent then got a false failure for a call that never ran, and the next controller could not recover the call. A restart must never fail work > - This pull request leaves that call unsettled for the next controller, the same as a call that was already in its handler > - The benefit is that a controller restart no longer turns an agent's pending tool call into a permanent error, and a CI test that failed on four pull requests now passes deterministically ## Linked Issues or Issue Description **What happened?** `src/live/runnerd-codex-transport-orphaned-tool-at-attach.test.ts` › `settles an earlier generation's in-flight tool call at attach even when that attach never finishes startup` failed in "ci / Verify Paperclip Runner (vitest package (3/3))" on 2026-11-15. In each job the first try and the CI retry both failed: - PR #21757, head `d04825d812`, job 132824209377 - PR #21713, head `7758d56719`, job 132681892394 - PR #21729, head `c63a05a013`, job 132732839971 Every failure is the same, at line 74, right after `detachControllerForRestart()`: ``` AssertionError: expected [ { …(8) } ] to deeply equal [] + "type": "semantic_tool.result", "status": "pending", + "result": { "error": { "code": "semantic_tool_not_dispatched", + "message": "The turn stopped before this operation was dispatched.", "retryable": false } } ``` **Expected behavior** When a controller detaches for restart, a tool call that it has not dispatched stays unsettled. The next controller recovers it at attach, the same as a call that is in its handler. **Steps to reproduce** 1. Hold the semantic tool dispatch at a gate after the Runner commits the tool input. 2. Call `detachControllerForRestart()`, then release the gate. 3. On `master`, the control-plane state has the `semantic_tool_not_dispatched` result above. It is byte-identical to CI (same command id and payload). The new test case `(still before dispatch at detach)` does this. **Root cause** This is a Runner defect, not a test timing problem. `#detachControllerOnce` sets `#closed`, resolves the closed signal and resolves the turn-start admission with `false`. A call in `#handleSemanticToolInput` that has not reached the handler then throws `SemanticToolNotDispatchedError`. The durable control plane turns that error into a journaled `semantic_tool_not_dispatched` result. That result is correct when the turn stops. It is false when the controller only detaches for restart, because the Runner and the turn continue. The old test waited only for the committed input, so on a loaded CI runner the detach sometimes came before the handler call. **Production impact** Yes. During a controller restart or deploy, an agent tool call in this window got a non-retryable "turn stopped" error after the Runner was adopted, and the next controller did not recover the call. ## What Changed - `durable-prp-control-plane.ts`: new `SemanticToolHandedOffError`. When the dispatcher rejects with it, the control plane journals no result. The input stays unsettled for the next controller. - `runnerd-codex-transport.ts`: the semantic tool callback goes through `#dispatchSemanticToolInput`. If the call ends as not dispatched after the transport detached for restart, it throws `SemanticToolHandedOffError`. A normal close still journals `semantic_tool_not_dispatched`. - `runnerd-codex-transport-orphaned-tool-at-attach.test.ts`: the test waits for the exact stage instead of the committed input. It runs two cases: the call is in its handler at detach, and the call is held before dispatch across the detach. Both then check the existing attach recovery and the final `completed` result. ## Verification - Red on `master`: the `(still before dispatch at detach)` case fails every run with the CI assertion and payload. - Green on this branch: `npx vitest run src/live/runnerd-codex-transport-orphaned-tool-at-attach.test.ts` passes 2/2. - Loop on macOS with 4 extra busy-loop processes (load average about 13 before them), 50 runs each: this branch 50/50 (100 test cases). The old `master` test was 50/50 locally too, so the local loop does not show the CI interleaving. The gate does. - All 36 runner test files that use `detachControllerForRestart` or `SemanticToolNotDispatchedError`: 1180/1182 pass with 3 workers under load. The 2 failures (`runnerd-codex-transport-settlement.test.ts` and `runnerd-acpx-source-reconciliation.test.ts`, `kill EPERM`) pass 36/36 when run alone. - `npx tsc -p tsconfig.json --noEmit` and `npx tsc -p tsconfig.test.json --noEmit` pass in `packages/paperclip-runner`. - The other two static-check flakes from the same day are test-only and are fixed in a separate pull request. ## Risks - Medium-low risk. This changes Runner restart behavior for one window. A call that was not dispatched now goes to the next controller's recovery (`recoverSemanticToolInput`) instead of getting a final error. That is the same path that already handles a call in its handler, so no new recovery code is involved. - If no successor ever attaches, the input stays unsettled. That is the same state as for a call that was in its handler at detach. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edit, GitHub CLI). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
7a9177af0f |
test(runner): remove coreutils and pid-file timing assumptions from static checks (#21787)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Paperclip Runner package has a "static checks" CI job that runs `node --test` script tests > - Two of these tests failed on unrelated pull requests on 2026-11-15 and passed on re-run > - Each failure costs a re-run of the failed jobs and delays merges, and it teaches people to re-run instead of read > - One test assumed that a copied `/bin/sleep` acts as `sleep` under any name. The other read a pid file before the child finished writing it > - This pull request removes both assumptions, and one more instance of the second pattern in the same test family > - The benefit is that these tests no longer depend on the runner image or on process scheduling ## Linked Issues or Issue Description **What happened?** 1. `scripts/stage-runner-binary.test.mjs` failed in "ci / Verify Paperclip Runner (static checks)" on PR #21765, head `98197c87e9`, job 132850068174: - `atomically publishes while the previous executable is running`: `AssertionError: 1 !== null` at line 124 (`old.child.exitCode`), after 77 ms. - `a source rebuild selects fresh dev bytes ...`: `Expected "actual" not to be strictly deep-equal` (line 192). - That job ran on a GitHub-hosted `ubuntu-26.04` image (20261111.82.6). The same tests pass on the RunsOn `ubuntu24` image, for example job 132822472729. 2. `scripts/materialize-kimi-distribution.test.mjs` › `setup cancellation physically stops an in-flight owned child` failed in the same job on PR #21758, head `40e38fc28a`, job 132822472729: `AssertionError: The real child did not start` after 149 ms. **Expected behavior** The tests pass on every supported runner image and on a loaded runner. **Steps to reproduce** 1. stage-runner-binary: in a Linux container, replace `/bin/sleep` and `/bin/echo` with one multicall binary that picks the utility from `argv[0]` and exits 1 for an unknown name (this is how uutils coreutils works on Ubuntu 26.04). Run `node --test scripts/stage-runner-binary.test.mjs`. On `master` both tests fail with the CI messages, 20 of 20 runs. 2. kimi: make the child create the pid file, wait 300 ms, then write the pid. On `master` the test fails with `The real child did not start` in about 125 ms, the same as CI. **Root cause** 1. On Ubuntu 26.04, `/bin/sleep` and `/bin/echo` are the same uutils multicall binary. The test copies it to `paperclip-runnerd` and runs it. The binary sees `argv[0] = paperclip-runnerd`, finds no utility of that name, and exits 1. The fresh-build fixture (a copy of `echo`) also has the same bytes as the old artifact (a copy of `sleep`), so the "fresh bytes" assertion fails too. 2. The child wrote the pid with `writeFileSync`, which creates the file and then writes it. The test loop broke on the first successful read. An empty read gave `Number("") === 0`, so the loop stopped and the assertion failed. The loop also had a fixed budget of 100 polls of 10 ms for a Node child to start. **Production impact** None. Only tests change. The staging code works with any executable. The Kimi setup cancellation code is correct: the deterministic repro shows that the child is stopped once the test reads a real pid. ## What Changed - `stage-runner-binary.test.mjs`: the test helper passes `argv0` to `spawn`. The copied `sleep` runs with `argv0: "sleep"`, so a multicall binary still acts as `sleep`. The fresh-build fixture is now a copy of `/bin/sh` and runs `-c "exit 0"`, so its bytes always differ from the old `sleep` artifact. - `materialize-kimi-distribution.test.mjs`: the child writes `pid.tmp` and renames it to `pid`, so a read never sees a partial file. The test waits until the pid is valid or the command settles. The 3 s command timeout and 5 s test timeout are unchanged. - `provision-kimi-package.test.mjs`: the wait for the `download-ready` marker continues until the child exits, instead of a fixed 1 s budget. The existing 5 s kill timer still bounds it. - No retries, skips or timeout changes. ## Verification - stage-runner-binary, Linux container (`node:24-bookworm-slim`) with a multicall `sleep`/`echo` built from a 15-line Go program that dispatches on `argv[0]`: - `master`: 0/20 pass, same two assertions as CI. - this branch: 50/50 pass. - this branch with stock coreutils, and on macOS: all 9 tests pass. - kimi setup cancellation: - delayed-write repro: `master` fails, this branch passes. - loop on macOS, `master` test without extra load: 59/60 (one natural failure). This branch: 60/60. - loop with 4 extra busy-loop processes (load average about 13 before them): `master` 100/100, this branch 100/100. - `node --test scripts/stage-runner-binary.test.mjs scripts/materialize-kimi-distribution.test.mjs scripts/provision-kimi-package.test.mjs`: 19 pass, 1 skipped (public download test, opt-in). - The other CI flake from the same day, `runnerd-codex-transport-orphaned-tool-at-attach.test.ts`, is a Runner defect. It is fixed in a separate pull request. ## Risks - Low risk. Only tests change. - The rebuild test now uses `/bin/sh` as the fixture for the fresh build. This file exists on every supported Linux and macOS runner. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edit, GitHub CLI, Docker). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>canary/v2026.1115.0-canary.15 |
||
|
|
d14a3b88b1 |
fix(native): release an idle warm session whose Runner is proven gone, and retry a failed idle close (#21782)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A warm agent session keeps its Runner alive in a sandbox between turns. The server closes idle sessions, and closes them again before it exits. > - When an idle close failed, the idle-timer path kept the session registered and never tried again, unless the sandbox lease was already inactive. > - The controller-restart path tried once more, but it accepted no proof that the session was already gone. It checked a destroyed sandbox only for a forced environment delete. > - In a release-swap test, one idle close failed with `runner_remote_process_identity_unavailable`. The session stayed registered until shutdown, the shutdown close failed again, and the sandbox was held until the host stopped it. > - This pull request treats a failed close as complete when the Runner is proven gone or the sandbox is stopped or destroyed, retries a failed idle close a bounded number of times, and logs which Runner check failed. > - The benefit is that a dead warm session is released, and the next failure names its cause. ## Linked Issues or Issue Description Refs #21627 **What happened?** In a zero-downtime release-swap test, a warm session's idle close failed 4 s after the server left new run starts. The failure was `runner_remote_process_identity_unavailable`. The idle-timer path kept the session registered with no retry. At SIGTERM, `closeIdleWarmNativeSessionsForRestart` tried again and failed the same way. The log showed "idle warm native sessions did not close" and the restart-chaos harness counted a warm-shutdown failure. The log did not say which Runner check failed, so the cause of the unreadable identity is not known. **Expected behavior** A close whose Runner is proven gone, or whose sandbox is stopped or destroyed, releases the session. A failed idle close is tried again a bounded number of times. The failure text names the Runner check stage and reason. **Steps to reproduce** 1. Keep a warm remote session idle. 2. Make its close fail with a `RemoteRunnerRecoveryError` (for example, `process_verification/process_not_running`). 3. Call `closeIdleWarmNativeSessionsForRestart()`. On master the result is `failed: 1`. ## What Changed - `server/src/services/native-runtime/native-session-executor.ts`: - New `warmCloseFailureProvesOwnerGone`. It uses existing proofs only: the close error's Runner diagnostic is `process_verification/process_not_running`, or a lease row for the sandbox has a termination receipt (`hasRemoteTerminationReceipt`) with state `destroyed`, or the warm run's own lease row has a receipt with state `stopped`. `pending_cleanup` and a lease without a receipt prove nothing. A lease read error counts as no proof. - `closeIdleWarmNativeSessions` (controller restart, environment delete, idle sleep) uses this proof for every failed close, not only for a forced environment delete. A released Runner detach failure is unchanged. - The idle-timer path uses the same proof. Its earlier status-only rule from #21504 (any lease status other than `active` or `retained`) is removed, because `pending_cleanup` is written before the provider stop. Without proof, it re-arms the timer and tries again, every 5 s, at most 3 times (`WARM_IDLE_CLOSE_RETRY_MS`, `WARM_IDLE_CLOSE_MAX_RETRIES`). The log line records the attempt number. - `describeWarmCloseFailure` appends the Runner check as `[stage/reason/lifecycle]` when the error carries one. - `server/src/services/native-runtime/native-session-executor.test.ts`: three tests. - "an idle warm session whose close fails with runner_remote_process_identity_unavailable and whose Runner is proven gone is released at controller restart" - "names the Runner check in a warm close failure and keeps an unproven owner registered at controller restart" - "retries a failed idle warm close a bounded number of times" - The #21504 idle-path test is now "forgets a warm owner whose idle close failed only once its sandbox has a stop or destroy receipt". A stop or destroy receipt releases the owner. `pending_cleanup`, `released` without a receipt, and `retained` keep it. - "reads the warm owner's own lease rows at controller restart" (a stop receipt for this run releases the owner; `pending_cleanup`, no receipt, or another run's receipt do not) ## Verification - `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts`: 1519 passed, 7 skipped. - Red on master (with only the two exported constants added so the file compiles): all three new tests fail. The first fails with `"failed": 1`. - `npx tsc --noEmit -p server`: no errors. ## Risks - A session is released without a checkpoint only when its Runner is proven gone or its sandbox is stopped or destroyed. The last saved checkpoint stays the recovery source, which is the same as the forced environment delete path today. - Environment delete and idle sleep now also accept the proof. Before, they accepted only a destroyed receipt during a forced delete. `pending_cleanup` is written before the provider stop, so it never counts as proof here. - The idle retry adds at most 3 more close attempts per failed session, 5 s apart. - A failed idle close whose sandbox stop is still pending now keeps its owner until a receipt lands or a successor retires it. Before, the status alone released it. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used in Claude Code with tool use (shell, file edit) and extended thinking. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
171e70c11d |
refactor: drop span and run-log event catalogs from the observability docs (#21779)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - `doc/observability.md` and `doc/run-log-events.md` described every OpenTelemetry span and every run-log event > - `rules/data-paths.mdc` told authors to update `doc/run-log-events.md` for each event, so both files only grew: 2,313 and 1,263 lines > - Each span and event is already defined next to the code that writes it, and its name is a literal string that a search finds in one step > - This pull request removes the per-span and per-event catalogs and keeps only what code does not say > - The benefit is two short docs that do not drift from the code, and no rule that makes them grow again ## Linked Issues or Issue Description Refs #21670, #21700, and #21743 (the same cleanup for other docs). **Issue type** Outdated or confusing documentation. **Where is the issue?** `doc/observability.md`, `doc/run-log-events.md`, and `rules/data-paths.mdc` **What's wrong?** The two docs repeated span names, span attributes, and event payloads that the code defines. Operator setup started at line 138 of `doc/observability.md`. The rule required a doc edit for every new run-log event. **Suggested fix** Keep operator setup and the Sentry privacy contract. Point to the code for span and event definitions. Remove the rule line that requires a doc entry per event. ## What Changed - `doc/observability.md` is now 619 lines. It keeps: how to enable tracing, `service.version` resolution, scope, how to enable Sentry, DSN delivery, environment attribution, privacy settings, server request data, the core of "Run failure context", browser data, fail-open behavior, the default capture set, and operator responsibilities. Kept text is unchanged except the intro and two sentences noted below. - Deleted from `doc/observability.md`: budget, Slack receipt, task detail, and browser tracing notes; controller preparation phases; native Runner spans; sandbox startup spans; sandbox duplex transport; shared skill preparation; invocation attribution; and about 500 lines of per-failure detail in "Run failure context". - Replaced that per-failure detail with one paragraph. It names the failures that `unexpected` mode keeps out of Sentry, and says the remaining diagnostic contexts are defined in `server/src/sentry.ts`. "Default capture set" now says the same. - `doc/run-log-events.md` is now 23 lines. It says what the run log is, where events are defined and appended, and keeps the payload storage note. - `rules/data-paths.mdc`: the run-log line now says to define an event where it is written, not in a doc. The span-allowlist line now says the allowlist is in the code that opens the span. The review levels and globs do not change. - Repointed links in two feature maps, two agent-access map files, `tests/runner-e2e/SUITES.md`, `doc/runner-api-tools.md`, and the Telemetry README. Updated the `honeycomb-tracing` skill and one code comment in `packages/adapter-utils/src/http2-bridge-server.ts`. ## Verification - A script built the new `doc/observability.md` from line ranges of the old file and asserts the first line of each range. - A link check on all changed Markdown files found 0 missing files and 0 missing anchors. The remaining inbound anchors (`#enabling-tracing`, `#sentry-error-monitoring`) still exist. - No test reads either doc. - `node scripts/check-maps.mjs` passes. - `node --test .github/scripts/tests/*.test.mjs`: 808 pass, 0 fail. - `pnpm verify` was not run. This worktree has no installed dependencies. The only code change is a comment. ## Risks - The privacy contract no longer lists each run-failure diagnostic context field. `server/src/sentry.ts` is now the list. A reviewer of a Sentry change must read that file. - No browsable list of span names or run-log event names remains. A search for the quoted name finds the producer. - `.greptile/files.json` still scopes `rules/data-paths.mdc` to both docs. Both files still exist, so it does not change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Claude Code desktop, with tool use. Earlier analysis in this session used Claude Fable 5.2. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [ ] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
531db3c10b |
perf(db): keep dedicated connections for the next burst (#21769)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - One server process runs every agent on one Node main thread, and each new database connection costs a TLS handshake, about 0.36 ms of SCRAM key derivation on that thread, and a type-fetch query > - `withDedicatedDbConnection` gives lock holders a private session; since #21693 it keeps at most 4 released sessions and closes the rest > - When the same burst of more than 4 dedicated holders comes back, every holder above 4 opens a new connection again > - A harness on embedded PostgreSQL measured 504 new connections per 1,000 uses for bursts of 8 and 760 for bursts of 16 > - This pull request keeps released dedicated clients for reuse while idle plus in-use clients stay within `DATABASE_DEDICATED_CONNECTION_MAX`; the driver idle timeout still closes their sessions when they are not used > - The benefit is 8 and 16 new connections per 1,000 uses for the same bursts, with no change to any pool maximum or to `databaseConnectionBudget()` ## Linked Issues or Issue Description **What is the problem?** A staging measurement showed `SET application_name='paperclip-runner-checkpoint'` at 3.17/s and `SET application_name='paperclip'` at 3.86/s with 7 live runs. This looked like connection churn. The investigation found two separate facts: 1. The `SET` statements are not new client connections. postgres.js sends `application_name` in the startup packet, never as a `SET` statement (`StartupMessage` in postgres.js `src/connection.js`). PgBouncer in transaction mode sends `SET application_name` itself when it gives a client transaction a server connection that a client with another `application_name` used last. With one shared server pool, every checkpoint transaction that lands after a main-pool transaction costs one `SET`, and the reverse. The rate follows checkpoint transactions, not connects. The checkpoint pool opens at most 3 connections per 1,000 saves (measured below). 2. Dedicated connections did churn by construction for recurring bursts of more than 4 holders (measured below). This pull request fixes that. **Steps to reproduce** Run `packages/db/src/connection-reuse-census.test.ts`. It counts `pg_stat_database.sessions` around 1,000 operations. **Expected behavior** A recurring burst reuses the sessions that the previous burst opened. ## What Changed - `withDedicatedDbConnection` keeps a released client idle when fewer than `DEDICATED_CONNECTION_IDLE_MAX` (4) are idle, as before, or when idle plus in-use clients stay within the dedicated limit (`DATABASE_DEDICATED_CONNECTION_MAX`, default 24). A client whose action failed, an ended client and clients after shutdown are still closed. - Each retained client keeps the main pool's `idle_timeout` (default 60 s), so its session closes after that idle time, as the main pool's do. The next use reconnects. - The worst case stays at `2 x DATABASE_DEDICATED_CONNECTION_MAX + 4`: a new client opens only when no client is idle, and a client is kept above 4 only while idle plus in-use is within the limit. `databaseConnectionBudget()` and every pool maximum are unchanged. - New `connection-reuse-census.test.ts` measures connections opened per 1,000 checkpoint saves and per 1,000 dedicated uses. - `dedicated-connection-pool.test.ts`: the burst test now checks that the next burst reuses all sessions and that they close after the idle timeout (1 s in the test). Before, it checked that at most 4 stayed open. - `doc/DATABASE.md` describes the new retention rule. ## Verification Connections opened per 1,000 operations, embedded PostgreSQL, `pg_stat_database.sessions` (load average about 10.5 on a shared machine): | Operation | Before | After | | --- | --- | --- | | Checkpoint saves, concurrency 1 / 7 / 24 | 1 / 3 / 3 | 1 / 3 / 3 | | Dedicated uses, steady concurrency 1 / 4 / 8 / 16 | 1 / 4 / 8 / 16 | 1 / 4 / 8 / 16 | | Dedicated uses, recurring bursts of 8 | 504 | 8 | | Dedicated uses, recurring bursts of 16 | 760 | 16 | - `pnpm --filter @paperclipai/db exec vitest run src/connection-reuse-census.test.ts src/dedicated-connection-pool.test.ts src/dedicated-connection-admission.test.ts src/client-options.test.ts`: 38 passed. The burst cases fail on `master` (504 and 760 against a bound of 8 and 16). ## Risks - After a burst, up to `DATABASE_DEDICATED_CONNECTION_MAX` dedicated sessions stay open for up to `DATABASE_IDLE_TIMEOUT_SECONDS` (60 s) instead of closing at once above 4. This stays inside the connection budget. With `DATABASE_IDLE_TIMEOUT_SECONDS=0` they stay open until shutdown, like the main pool. - Retained sessions keep the existing rule: actions keep session state transaction-scoped. - The PgBouncer `SET application_name` statements stay. Removing them needs one shared `application_name` for all pools, which removes the per-pool names that operators and tests use. That is a separate decision. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
8f1d39da79 |
perf(server): cache the instance settings row per process (#21768)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - One server process runs every agent on one Node main thread, and
each database statement costs main-thread time
> - The `default` instance settings row is read on hot paths: live run
event delivery, run log redaction, the stranded-task recovery check,
runner tool calls, adapter routes and run preparation
> - A staging measurement with 7 live runs showed the full-row settings
read at 10.24 statements per second, and 3.5 to 5.4 per second when idle
> - The row changes only when an admin or a boot step writes it, so
almost every read returns the same value
> - This pull request adds a per-process read-through cache for that
row, with in-process invalidation on every write and a 2 second age
bound for changes made by another process
> - The benefit is at most one settings read per 2 seconds per server
process, whatever the load
## Linked Issues or Issue Description
**Is your feature request related to a problem?**
The full `instance_settings` row read (`select id, singleton_key,
default_environment_id, general, experimental ...`) runs at 10.24
statements per second on staging with 7 live runs, and 3.5 to 5.4 per
second at idle. It scales with run events and recovery checks, not with
settings changes.
**Describe the solution you would like**
Cache the row per process. Drop the cached row when a settings write
commits, in this process and, through the existing work-signal
notification, in the other process during a deploy overlap. Bound the
age of a cached row to 2 seconds, so any other change (a direct database
edit, a second process outside an overlap) takes effect within 2
seconds.
**Describe alternatives you have considered**
A database trigger with LISTEN/NOTIFY would also catch direct edits, but
it needs a migration, and work notifications cross processes only during
a deploy overlap today. The existing signal plus a 2 second bound keeps
the change small.
## What Changed
- New `server/src/services/instance-settings-cache.ts`: a read-through
cache keyed by the database client. Concurrent misses share one read. A
cached row is at most `INSTANCE_SETTINGS_CACHE_TTL_MS` (2000 ms) old,
measured from the start of the read. Every caller gets its own copy.
Reads through a transaction handle skip the cache.
- `instanceSettingsService` `get`, `getGeneral` and `getExperimental`
use the cache. Write paths (`update`, `updateGeneral`,
`updateExperimental`, and the feedback vote save) still read the row
fresh.
- Every settings write publishes an `instance-settings-changed` work
signal through the existing work-signal channel
(`@paperclipai/db/work-signals`). Inside a transaction it is the first
signal of the write, so its `pg_notify` payload goes out before the
other wakes of the same commit (for example `status-card-evaluation`).
This process drops the cached row when the write commits. During a
deploy overlap, when work notifications cross processes, the other
process drops its cached row on receipt, before a worker woken by the
same write reads the settings. A reconnect of the notification listener
also drops the row.
- `fillUnsetInstanceDefault` writes through `update({ db: tx })`, so its
signal is sent when its transaction commits.
- A read that starts before an invalidation never stores its result, so
an old value can not come back after a write.
- The test setup clears the cache before each test, because many tests
write the row directly.
Hot callers of the cached row:
- Live run event delivery: `live-run-event-redaction.ts`
(`censorUsernameInLogs`), once per delivery batch.
- Run output and run lifecycle redaction: `heartbeat.ts`
`getCurrentUserRedactionOptions`, used by `run-output.ts`,
`run-lifecycle.ts`, `run-completion.ts`.
- The stranded-task recovery check: `recovery/service.ts` reads
`getExperimental().enableAgentChat` once per checked conversation issue.
- Runner tool calls: `paperclip-runner-tool-authority.ts`
(`enableIsolatedWorkspaces`, `enableExternalObjects`).
- Adapter routes: `app.ts` (`enableNativeRunner`, `enableOpenAiDot`) per
request.
- Run preparation: `heartbeat.ts` (`instance_settings` preparation
step), `queued-run-line.ts`, `queued-run-dispatch-liveness.ts`,
`run-control.ts`.
- Issue and comment paths in `issues.ts` (`censorUsernameInLogs`,
`enableAgentChat`).
## Verification
- `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/instance-settings-cache.test.ts` (embedded PostgreSQL):
- 150 hot reads (`getGeneral`, `getExperimental`, `get` from new service
objects) after one warm read: 0 statements. Without the cache the same
test measures 150 statements (red run).
- 20 concurrent misses: 1 statement.
- `updateExperimental` and `updateGeneral` are visible at once to a new
read in the same process.
- `fillUnsetInstanceDefault` (transaction write of the default
environment) is visible at once after it returns.
- A change written by a second client (another process) becomes visible
within 2000 ms plus a 500 ms test margin.
- A caller that changes the returned object does not change the cached
row.
- A read through a transaction handle sees the transaction's uncommitted
write, two such reads issue 2 statements, and the cache does not keep
the write after rollback.
- A read that started before an invalidation is not stored.
- A received `instance-settings-changed` notification from another
process drops the cached row (next read issues 1 statement).
- `updateExperimental({ enableStatusCards: true })` sends
`instance-settings-changed` as its first notification, before
`status-card-evaluation`; `updateGeneral` sends it alone.
- The server tests that use instance settings or write the row directly
(230 files): see the run notes in the first comment.
## Risks
- A direct database edit, or a change from a second process outside a
deploy overlap, takes up to 2 seconds to apply in this process. Before
this change it applied at the next read. This includes access flags such
as `enableOwnerInstanceAdmin` (read per cloud-tenant request) and the
default environment.
- A change made through the settings service applies at once in this
process, and during a deploy overlap at once in the other process.
- A new code path that writes the `default` row must call
`announceInstanceSettingsChange`, or its change applies within 2
seconds.
- No schema change.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, test runs).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (no
user-facing setting or command changed)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
3706f520fd |
test(scaling): keep ramping through recovered runs, read spend from cost records, and wait for termination before deleting agents (#21776)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The scaling effort uses a real-agent ramp harness (`scripts/scale-agents-ramp.ts`, #21695 and #21757) to load a hosted test instance through the product API > - Its first run with agents that really ran (10 agents, 105 tasks, 106 runs) stopped on one failed run, recorded $0 model spend, and left cleanup errors > - Read-only inspection of the instance showed three harness faults. The failed run had a same-run bootstrap retry scheduled 33 s later, but the harness stopped and cleaned up first. Run rows carry no cost, but cost records do (63.7 cents for the run's agents). Terminated agents leave the company agent list, so the harness treated them as done and sent deletes that Paperclip refused with 409 > - These faults make the ramp stop too early, make its spend cap inert, and make cleanup report the wrong cause > - This pull request keeps the ramp going through runs that Paperclip recovers by itself, reads spend from cost records, records board latency per route, and makes cleanup wait for termination and name what it left behind > - The benefit is that the next ramp grades the product, not the harness, and its spend cap works ## Linked Issues or Issue Description Refs #21695 Refs #21757 **What happened?** The ramp stopped at step 1 with `failed_runs_zero failed (observed 1)`. The run failed with `runner_ingress_bind_conflict` before its provider session opened. Its run events show `same-run resume persisted` with `recoverable: true` and `nextAttemptAt` 33 s later. The harness stopped within 9 s and paused the agents, so the retry was blocked (`native_retry_admission_ineligible`). The result showed `Recorded model spend: $0`. The harness read `costUsd` from run rows, which is always null; `GET /api/companies/<id>/costs/by-agent` holds 63.71 cents over 104 cost events for the 10 agents. Cleanup reported two `DELETE /api/agents/:id returned 409: Complete termination before deleting the agent`. Those agents were still `lifecycleState: terminating`; the harness waited on the company agent list, which hides terminated agents, so it treated them as done. A later `--cleanup-only` changed nothing because it also found agents only through that list. The board API p90 failed (765 ms), but the result had no per-route timings to say which request was slow. **Expected behavior** A failed run whose task finishes by itself is counted, not a stop. A failed run whose task does not finish within a grace stops the ramp. The share of failed runs is graded at the step end. Spend comes from Paperclip's cost records. Cleanup deletes an agent only after termination completes, and names the agents and sandbox leases still pending. **Steps to reproduce** 1. Run `node --test scripts/scale-ramp/scale-ramp.test.ts` with the new fake and tests and the `origin/master` `ramp.ts`, `cleanup.ts`, `criteria.ts` and `report.ts`. 2. 16 tests fail, among them every new test (recovered run, unrecovered run, spend from cost records, spend cap, board per route, termination wait, still-terminating agents and `--cleanup-only`, leases released during the wait). 3. With this change, all 60 pass. **Paperclip version or commit** Instance on `2026.1106.0+2564.git.3244780bd`; harness from #21757. ## What Changed - `criteria.ts`: replace `failed_runs_zero` with `failed_runs_unrecovered_zero` (immediate) and `failed_run_rate` (at the step end). Step metrics add `runsEnded`, `runsFailedRecovered`, `runsFailedUnrecovered`, `runsFailedPending` and `board.byRoute`. - `ramp.ts`: a run counts as failed once it was seen `failed` or `timed_out`. It is recovered when its task reaches `done`. It is unrecovered when the task rests not done, needs Retry, or has no live run for `--recovery-grace-minutes` after the failure. A task whose interruption has an automatic retry scheduled (`state: retry_scheduled`, `nextRetryAt` or `retryWait`) does not count as needing Retry. Spend comes from `costs/by-agent` for the run's agents on each poll, and each step reports the spend in its window. Board probes keep a distribution per route. - `config.ts`: add `--max-failed-run-percent` (default 2) and `--recovery-grace-minutes` (default 15). - `cleanup.ts`: read agents by id as well as from the list, so terminated agents in the state file are found. Wait for `lifecycleState` `terminated` or `rejected` by id before a delete. Report `agentsStillTerminating`, `activeSandboxLeaseIds` and `sandboxLeasesReleasedDuringWait`, and name each lease still active with its run and acquire time. - `report.ts`: the summary shows recovered, unrecovered and pending failed runs, board latency per route, and each cleanup error. - `fake-paperclip.ts`: run rows carry no cost; `costs/by-agent` returns cents per agent. Optional failure of the n-th run, a scheduled recovery run, a termination delay (terminated agents leave the list and a delete answers 409 until termination completes), and leases that stay active after their run. - `doc/plans/2026-11-13-thousand-concurrent-agents.md`: the runbook describes the failed-run criteria, the spend source and the cleanup behavior. ## Verification - `node --test scripts/scale-ramp/scale-ramp.test.ts`: 60 of 60 pass. - Red check: with the new fake and tests but the `origin/master` `ramp.ts`, `cleanup.ts`, `criteria.ts` and `report.ts`, 16 tests fail, including all 8 new ones. - `tsc -p scripts/scale-ramp/tsconfig.json`: no errors. - Evidence from read-only GETs on the test instance: run `f1892d43` events (bind conflict, scheduled same-run retry, blocked after the pause), run rows (`costUsd` null on all 106), `costs/by-agent` (63.71 cents for the 10 agents), agent rows by id (`terminating` while absent from the list), lease rows. - Not run: a new real ramp (needs the owner's go-ahead). ## Risks - Low risk. The change touches only the scale-ramp harness, its fake, its tests and one runbook section. No product code changes. - A failed run no longer stops the ramp at once. An unrecovered one stops it after the grace (15 min by default), so a broken step can run up to that long. The step-end failed-run rate still fails a step with more than 2% failed runs. - Spend now needs cost read access for the board token. If `costs/by-agent` cannot be read, the result reports spend as not measured and the spend cap does not trigger. - Cleanup can now take up to 5 more minutes while it waits for termination by id. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, read-only HTTP GETs to the test instance). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
2b23119ed2 |
fix(native): keep a warm follow-up that inherited its predecessor's process identity waiting on the same run (#21774)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A deploy swaps servers with an overlap. A warm agent session stays with the outgoing server that ran its last turn. > - #21697 makes a follow-up wait while that outgoing server is still alive, instead of starting a second Runner over its session. > - Runtime selection copies the warm predecessor's process fields (`processPid`, `processGroupId`, `processStartedAt`) onto the follow-up and records `inheritedProcessIdentityRunId`. > - The wait records itself with `deferNativeControllerRecovery`. That function refused any run with process fields, so it refused every warm follow-up that the wait stopped. The run failed with `native_controller_recovery_wait_authority_changed`. > - This pull request lets the deferral accept an inherited identity, and clears those inherited fields in the same transaction so the usual resumption claim can pick the run up. > - The benefit is that a warm follow-up in a deploy overlap stays on the same run and does not fail. ## Linked Issues or Issue Description Refs #21697 **What happened?** In a zero-downtime release-swap test, a follow-up comment arrived 0.25 s after its predecessor finished on the outgoing server. The follow-up run started on the incoming server 1.09 s later. `waitForLiveWarmSessionOwner` found the outgoing server alive and asked to wait. `deferNativeControllerRecovery` then refused with `native_controller_recovery_wait_authority_changed`, because the run carried the process fields that runtime selection copied from its warm predecessor. The run failed after 2.7 s with `native_continuation_requires_reconciliation`. Held recovery started a fresh run. The task finished, but the run history shows one failed run. **Expected behavior** A warm follow-up whose only process identity is inherited from its predecessor waits on the same run. The resumption owner claims it at the deadline. A run whose own process has started is still refused. **Steps to reproduce** 1. Run a warm remote agent turn on server A. 2. Switch new run starts to server B while A still serves. 3. Post a follow-up within about 1 s of the turn's finish. 4. The follow-up fails with `native_controller_recovery_wait_authority_changed`. ## What Changed - `server/src/services/native-runtime/native-controller-recovery.ts` (`deferNativeControllerRecovery`): - Process fields refuse the deferral only when they are the run's own. Fields count as inherited when `runnerProfileJson.inheritedProcessIdentityRunId` is set and the run has no own `native.process_start_requested` or `native.process_identity_recorded` event. `run-cost-accounting.ts` uses the same proof. - With an inherited identity, the same transaction clears `processPid`, `processGroupId` and `processStartedAt`. It keeps `inheritedProcessIdentityRunId`. `claimNativeSessionResumptions` requires those fields to be null. - `server/src/services/native-runtime/native-controller-recovery.integration.test.ts` (real Postgres): - New: "keeps a warm follow-up that inherited its predecessor's process identity on the same run until the outgoing owner's deadline". - Two new cases in "does not publish an admission wait after authority changes": `own-process` (process fields, no marker) and `inherited-then-own-process` (marker plus an own process-start event). Both stay refused and keep their fields. ## Verification - `cd server && npx vitest run src/services/native-runtime/native-controller-recovery.integration.test.ts -t "inherited its predecessor|does not publish an admission wait|unstarted follow-up"`: 9 passed. - Red on master: the new test fails with `Error: native_controller_recovery_wait_authority_changed`. The own-process cases pass on master and with the change. - Green: the run is `retryable_failure` with `nextAttemptAt` at the deadline, its process fields are null, no claim happens before the deadline, and one claim happens at the deadline. ## Risks - The follow-up now waits instead of failing. The wait ends when the outgoing server saves a suspended checkpoint, when it exits, or at the predecessor's finish plus its idle window plus 2 minutes. In the observed case, the outgoing server's idle close could not save its checkpoint (`native_runner_checkpoint_owner_lost`), so the wait would end only when that server exits. See the follow-up note below. - An own process is still refused. The proof uses the run's own process-start events, the same proof as run cost accounting. **Follow-up note (not in this PR):** The outgoing server's checkpoint write checks the task's current run. A native successor that is parked in this wait is not one of the accepted successor kinds, and the idle close is not retried. So the wait does not end early from a checkpoint. In the observed timeline, the wait is about 225 s (until the outgoing server exited). The worst case with the default 5-minute idle window is about 425 s. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used in Claude Code with tool use (shell, file edit) and extended thinking. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
b9cf64f5bf |
test(server): check the agent row lock with FOR SHARE so lock checks cannot block each other (#21775)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Each master commit must pass Cloud readiness before its canary and image publish > - `agent-file-restore-locking.test.ts`, added in #21694, failed master Cloud readiness run 43712563988 on 2026-11-15, so that commit's canary did not publish > - The test reported the agent row as locked during the download, but its own sampler measured 0 ms of product lock time > - The cause is in the test: its two lock checks used `FOR UPDATE NOWAIT` on one pool and could block each other > - This pull request checks the lock with `FOR SHARE NOWAIT`, which still detects the product's `FOR UPDATE` but never conflicts with another check > - The benefit is that the test fails only when the product really holds the lock ## Linked Issues or Issue Description **What happened?** `downloads a stale projection without holding the agent row lock` failed in master Cloud readiness run 43712563988, attempt 2: `expected [{ objectKey, agentRowLocked: true }] to deeply equal [{ objectKey, agentRowLocked: false }]`. The test's metric line showed `lockedMs: 0`. **Steps to reproduce** 1. Make each sampler check hold its `FOR UPDATE NOWAIT` lock for 50 ms (to make the overlap certain). 2. Run `pnpm exec vitest run src/__tests__/agent-file-restore-locking.test.ts -t "stale projection"` in `server/`. 3. The test fails 3 of 3 times with `agentRowLocked: true` and `lockedMs: 0`, the same as CI. **Expected behavior** Only a lock held by the product makes the check report "locked". Refs #21694 ## What Changed - `agentRowLocked()` uses `FOR SHARE NOWAIT` instead of `FOR UPDATE NOWAIT`. The product takes `FOR UPDATE` on this row (`agent-file-store.ts`, `agent-instruction-revisions.ts`), which conflicts with `FOR SHARE`. Two `FOR SHARE` checks do not conflict. - New test: three concurrent checks all report unlocked, a `FOR UPDATE` held on another connection is reported locked, and the row reads unlocked again after release. ## Verification - Red: with the forced 50 ms overlap and the old `FOR UPDATE` probe, the test failed 3 of 3 runs with the CI error. - Green: with the same forced overlap and `FOR SHARE`, 4 of 4 tests passed in 3 of 3 runs. - Final file (no forced overlap): `pnpm exec vitest run src/__tests__/agent-file-restore-locking.test.ts` in `server/`: 5 of 5 tests passed in 5 of 5 runs. ## Risks - Low risk. Test-only change. The new test makes sure the check still detects a real `FOR UPDATE` lock. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
62033b9994 |
fix(qa): schedule recovered cleanup promptly and wait for task panels (#21660)
## Thinking Path > - Paperclip lets people manage AI agents and their saved work. > - QA journeys check browser controls and the cleanup of owned agent resources. > - A recovered workspace queues sandbox destruction after its files are saved. > - The cleanup queue treated that first attempt as a retry and waited five minutes. > - Agent termination correctly waited for destruction, so QA cleanup failed first. > - This PR runs that first cleanup attempt promptly and keeps the existing safety gates. > - It also waits for task-panel transitions before the browser acts again. ## Linked Issues or Issue Description **What happened?** The full scheduled QA smoke completed with 23 passed and 1 failed. Stop/redirect passed all eleven behavior checks but failed cleanup. The cancelled run had a saved workspace and a bound stop receipt before agent termination. Its sandbox was destroyed about five minutes later. The shared cleanup schedule applied the retry delay before the first destruction attempt. The browser regressions also found that the current panel helper can consume all four actions while an opener remains visible during panel mounting. **Expected behavior** A recovered workspace's first cleanup attempt runs on the next eligible worker pass. A failed attempt retains normal backoff. Agent termination still waits for completed resource cleanup. Task-panel navigation waits for the next controls before it acts again. **Steps to reproduce** 1. Queue recovered-workspace destruction after the cleanup worker has become idle. 2. Read the next cleanup deadline and advance one normal worker pass. Before this fix, the deadline is five minutes late and no teardown runs. 3. Open task document controls with a delayed side-panel transition. Before the panel fix, both collapsed-panel controls fail. Refs #19224 and #19292 for retaining stop proof during workspace recovery. This change keeps the full cleanup requirement for agent termination. Current master supplies the shared task-panel helper; this branch uses it and retains additional browser regressions. ## What Changed - Use one shared first-attempt predicate for the cleanup deadline, sweep selection and ownership report. It now includes recovered workspace cleanup bound to the same run. - Preserve worker spacing, in-flight claims, ownership and saved-file checks, retry delays and attempt limits. - Wait for panel controls after opening the panel. Wait for the selected tab or menu action to finish. - Add real PostgreSQL cleanup regressions, keep the Chromium controls, and document the cleanup timing. ## Verification Tested on Linux x86_64, Node v25.9.0. Local commit `7c8e0f12aa4b19aaf53669141ab3299ce76db963` and published head `892c5dd4b4638796a138400d8bbe30aee19c71dd` have the same tree, `d135a98bdff58e3320cdc83976ffe1cf47f48b66`. - Red: both new real-PostgreSQL cleanup tests failed on the old code. The deadline was five minutes late and the first normal sweep made zero teardown calls. - Green: 87 cleanup, ownership and lifecycle tests pass. The restored-work case stays unconfirmed until the real cleanup worker destroys the lease; failure still backs off and the worker becomes idle again. - Red after master integration: both delayed document-panel browser cases failed. Green: all 18 Chromium support checks pass. - 77 journey contract tests pass. - Product E2E typecheck passes with the documented low-memory path. The initial ordinary attempt was killed by the local memory limit and is not counted as a pass. - The only follow-up after the passing checks removes new code comments to follow the contribution guide. Runtime and test behavior are unchanged. Final-head [CI run 43714742904](https://github.com/paperclipai/paperclip/actions/runs/43714742904) completed successfully at this exact head. All 63 PR checks are terminal: 58 success, 2 neutral and 3 conditional skips; no failures or pending checks. Typecheck, build, tests and the final verify gate pass. Greptile and Paperclip Review both score this head 5/5 with no actionable findings; both historical inline threads are resolved. - Hosted QA recovery is still unverified. No extra model run, merge, deployment or QA setting change was made. ## Risks Recovered workspace destruction starts sooner. Existing producer, file-preservation, ownership, claim and provider checks still decide whether teardown is allowed. Failed attempts retain the existing cooldown. Agent termination and purge gates are unchanged. The panel helper waits for visible controls and still fails when controls or saved values are absent. The PostgreSQL integration uses a controlled provider boundary. It does not prove a new live Daytona or deployed QA pass. That requires the fixed revision on QA and a later normal scheduled smoke. ## Model Used OpenAI GPT-6-based Codex, with tool use and code execution. The host does not expose a more specific model ID or context-window size for this run. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub references) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
afca556728 |
fix(github): clarify configured review task completion (#21748)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - GitHub bots publish reviews and report the result of their assigned task. > - A GitHub check and a Paperclip task have separate completion rules. > - The current prompt explains how to wait, but gives little completion guidance. > - A bot can keep a complete review in review while it waits for CI or merge that its assignment does not require. > - This pull request makes the configured completion rules clear in GitHub prompts. > - The benefit is a finite review task with an accurate status and publication evidence. ## Linked Issues or Issue Description Related change: Refs #17318 keeps advisory review waits separate from human attention. This change adds completion guidance in the same GitHub prompt owner. **What happened?** A bot can publish a complete current-head 5/5 review and still wait for CI or merge. Manual request prompts also lack the task completion guidance used by automatic reviews. **Expected behavior** When the configured assignment allows completion at 5/5, a complete published review can finish Done. CI and merge are completion conditions only when the assignment requires them. Explicit tracking rules still apply. **Steps to reproduce** 1. Configure an agent to finish after a complete published current-head 5/5 review. 2. Start a GitHub review with CI results unavailable. 3. Inspect the generated prompt. It explains a response wait but does not explain this completion case or publication safeguards. 4. With this change, automatic, mention, and comment prompts carry the same explicit guidance. **Paperclip version or commit** Base commit: `83541e89ad3b90740ffc61a32c396dddb5834526`. **Deployment mode** Built from source. The prompt change applies to GitHub bot tasks. ## What Changed - Add shared guidance within the GitHub prompt module for configured completion at 5/5 and finite discussion replies. - Require a complete assessment, confirmed publication, and the current head in the 5/5 guidance. - Preserve explicit CI, merge, and below-5 tracking rules. Keep the shared status arbiter and check thresholds unchanged. - Add six prompt cases. State that CI is unavailable in the existing lifecycle completion case. - Document the configuration example and its limits. ## Verification - Six new prompt cases failed before the change. - `pnpm exec vitest run server/src/services/chat-github-review-policy.test.ts`: 40 passed. - `pnpm exec vitest run server/src/services/native-runtime/external-chat-wait.integration.test.ts -t 'settles a published'`: 9 passed. - `pnpm exec vitest run server/src/services/native-runtime/external-chat-wait.integration.test.ts -t 'inherits.*review|published.*guard|unknown publication|pending publication|changed head|inherited.*review'`: 5 passed. - `git diff --check` passed. - `pnpm verify` returned exit 2. The connector document requires broader CI coverage. - Local preflight installed dependencies and built shared and plugin SDK packages. It stopped because Cargo is unavailable. Full local typecheck, build, and test suites were not run. - [Greptile reviewed this commit](https://github.com/paperclipai/paperclip/pull/21748#issuecomment-6894002401) at 5/5 with no actionable findings. Its check passed for `36dd146cdc00b0f3d7f6fc2971f24166ef0f6bf5`. - [CI](https://github.com/paperclipai/paperclip/actions/runs/43711597984) passed for `36dd146cdc00b0f3d7f6fc2971f24166ef0f6bf5`. Typecheck, build, Rust/Runner, all server tests, and all eight E2E test groups passed. The final [CI gate](https://github.com/paperclipai/paperclip/actions/runs/43711597984/job/132808715297) passed at 12:17:58 UTC on 2026-11-15. - At 12:23 UTC, all 61 check records were complete: 57 passed, 1 was neutral, and 3 were skipped. Snyk also passed. No checks failed or remained pending. ## Risks This change gives prompt guidance. It does not enforce a score-to-status rule. Custom assignments can still require CI, merge, or later review. The focused tests check prompt content and existing lifecycle behavior; they do not prove every live agent or webhook journey. ## Model Used OpenAI GPT-6 through Codex, with reasoning, tool use, and code execution. This session does not expose the exact model variant or context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in this PR with the bug report fields - [x] I have not referenced internal or instance-local Paperclip issues or links - [x] My public branch name describes the change and contains no internal ticket id - [x] I have run focused tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
e6562755e1 |
fix(issues): cover a blocked issue whose own run is queued after its blockers finish (#21770)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A parent task is blocked on its subtasks. When the last subtask is done, Paperclip queues a run for the parent to continue. > - The board marks a blocked task "needs attention" when no live work covers it. The check looks at the blockers, then at each blocker's runs. > - When every blocker is done, the check returned "needs attention" at once. It did not look at the parent's own queued or running run. > - In a release-swap test, a parent showed "needs attention" for about 12 minutes while its own continuation run waited in the queue. Nobody had to do anything. The state cleared by itself. > - This pull request makes the check return "covered" when the blocked task has its own queued or running run. > - The benefit is that people see no false "needs attention" while Paperclip already continues the work. ## Linked Issues or Issue Description Refs #21731 **What happened?** In a zero-downtime release-swap test, a blocked parent task had two subtasks. Both subtasks were done. Paperclip queued the parent's "children completed" run, which waited about 12 minutes for a free agent slot. During that time, `blockerAttention` was `needs_attention` / `attention_required`, with 0 unresolved blockers and the parent itself as the terminal blocker. The board showed the parent in the blocked inbox. Then the run started, finished, and the parent became done. **Expected behavior** A blocked task whose own run is queued or running is covered by live work. It must not ask a person for attention. **Steps to reproduce** 1. Create a blocked parent task with one or more subtasks that block it. 2. Mark every subtask done. 3. Keep the parent's own run queued (for example, when the agent has no free slot). 4. Read the parent from the issue list. `blockerAttention.state` is `needs_attention`. ## What Changed - `server/src/services/issues.ts`: in `readIssueBlockerAttentionBatch`, when a blocked root has no unresolved blockers, the function now checks the active-issue set that it already built. If the root has its own queued or running run (or an active wake), it returns `covered` with `blockingTreeLive: true`. Otherwise it returns `needs_attention` as before. - The change adds no query. The active-issue set already includes the roots, through their `executionRunId` and their wake requests. - `server/src/__tests__/issue-blocker-attention.test.ts`: two tests. - "classifies a blocked parent whose children are all done as covered while its own children-completed run is queued" - "keeps a blocked issue with no unresolved blockers and no active run as needs_attention" ## Verification - `cd server && npx vitest run src/__tests__/issue-blocker-attention.test.ts`: 29 passed. - Red on master: the first new test fails with `"state": "needs_attention"`. Green with the change. - The second new test passes on master and with the change (negative case). ## Risks - Low risk. The change only affects a blocked task with zero unresolved blockers. That task now reads as covered while its own run is queued or running. When the run ends without a status change, the task reads as `needs_attention` again. - The covered result has no reason (`reason: null`), because no blocker covers it. The UI label is then "Blocked", not "waiting on 0 sub-tasks". ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used in Claude Code with tool use (shell, file edit) and extended thinking. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
5b750fdf25 |
feat(ui): add foldable sidebar subtasks with bounded refreshes (#21649)
## Thinking Path > - Paperclip helps people manage AI agents for work. > - The sidebar helps people return to recent tasks. > - Recent parents need foldable child tasks. > - Busy companies must not refresh every child group for unrelated work. > - Exact unread counts must preserve personal reads and task access. > - This pull request adds child groups and bounds their refresh work. > - People can scan subtasks without repeated background requests. ## Linked Issues or Issue Description **Subsystem affected** Task navigation in the standard and Classic sidebars. **Problem or motivation** Recent parent tasks did not show their subtasks. The first implementation refreshed every previously viewed group. One unrelated update caused 52 child-page requests and one summary request after viewing 50 groups. Exact child counts also repeated privacy and unread checks per child. **Proposed solution** Show direct subtasks with independent folding and ten-row pages. Refresh affected groups and summary batches. Stop offscreen reads. Bound child-request concurrency. Aggregate viewer facts once and preserve the existing inbox unread rule. **Roadmap alignment** This extends existing task navigation. The existing feature PR is reused for its performance repair. No new PR duplicates it. ## What Changed - Add shared parent groups to both sidebar layouts. Preserve Classic row actions and active-child visibility. - Keep unread state personal to each task. Folding and paging do not mark tasks read. Counts include readable children beyond ten. - Target live refreshes with authorized current and previous parent IDs. Reuse parent metadata from the narrow run-binding read. Resolve fallback lookups once per delivery batch. Keep the previous summary fallback for older servers. - Limit child requests to three at a time. Cancel active and queued reads when groups leave the viewport. - Discard extra cached pages on Show less and collapse. - Compute exact navigation counts in one SQL statement. Materialize readable candidate IDs once. Use indexed participation sources and conservative unread candidates before the shared inbox rule. - Skip recursive privacy walks for standalone open tasks without a project. Keep the recursive rule for all restricted or inherited shapes and all delegated principals. - Add request-count, cancellation, reparenting, read-receipt, inbox-parity and privacy regressions. Include a disposable PostgreSQL benchmark. - Update product and API documentation. Retain the production Storybook states. ## Verification - Published head: `4148861437c1b6b635479ad92509dbc71155a232`. Its tree matches local `04be9d66d481d9bcd1c4d6158acf063a1e65ac7e`: `a23533e93d3d155918f0254d296a7fd2f0c7c12f`. - Rebased on master `ca23330c69b1f2b25bdadfcbda68317db97deaef`. Remote publication preserves prior feature commits. - The request regression failed before the fix and passes after it. An unrelated event now sends zero child requests. Current servers also avoid the unrelated summary request. Old-server events keep one summary fallback. - A relevant update after Show less reads one page. Reparenting refreshes two affected groups. Offscreen reads stay at zero and resume once on return. Active and queued cancellation tests pass. - The same synthetic authenticated HTTP workload with 50,000 read children improved from 767 ms to 92.84 ms median. Five warm samples per case. The all-unread case improved from 463.85 to 351.65 ms. Untouched children improved from 208.01 to 131.09 ms. The old-receipt case improved from 799.70 to 245.80 ms. These local measurements are not a production capacity guarantee. - Focused PostgreSQL routes, inbox-parity, privacy and WebSocket tests passed: 38 tests. - All 325 affected UI tests passed. All 38 focused PostgreSQL, privacy and WebSocket tests passed. UI typecheck and low-memory server declaration checks passed. - All 41 Chromium Storybook checks passed. They cover 20 states in both themes and keyboard, folding, paging and unread behavior. They use production components with HTTP fixtures. - Both new review findings failed as regressions before their fixes and pass afterward. Explicit child reads include waiting external conversations. Deleting a child publishes a separate parent-authorized update without deleted-child content. Both review threads are resolved. - `pnpm verify --explain` returned exit 2 because product documentation requires broader CI. [The prior CI run](https://github.com/paperclipai/paperclip/actions/runs/43711489306) found type errors in the new test fixtures. The same errors failed locally. The fixture fix passes `node server/scripts/typecheck-layers.mjs --layer test-tree-0` and all 21 issue summary route tests. Production code is unchanged. That CI run then found a live-delivery read regression. The existing test failed locally with eight statements against its six-statement budget. A new PostgreSQL regression also found three parent lookups for two same-batch calls. Run delivery now reuses parent metadata from its narrow run-binding query. Fallback lookups resolve to one promise. Both regressions pass without increasing the budgets. All 74 targeted route, WebSocket and privacy tests pass. The affected below, above and test-tree-0 typecheck layers pass. [CI attempt 2](https://github.com/paperclipai/paperclip/actions/runs/43715541480/attempts/2) passes on the published head. All 62 check runs are terminal and clean: 57 succeeded, two were neutral and three were skipped. Typecheck, build, policy, all general and serialized server suites, E2E and the [final verify gate](https://github.com/paperclipai/paperclip/actions/runs/43715541480/job/132829783231) pass. The separate security status passes. Attempt 1's server shard 7/12 reached its 20-minute job limit without a test verdict. One retry of that job passed without a source change. [Greptile completed successfully on this head](https://github.com/paperclipai/paperclip/runs/132816163906) with a [5/5 score](https://github.com/paperclipai/paperclip/pull/21649#issuecomment-6883916606). All five review threads are resolved. The final CI gate is complete. - A full local Storybook build exceeded available memory while server typechecking ran. Browser verification uses the current-source development server. Rebuilding stale Runner and SDK declarations resolved the initial server check errors. The first browser attempt lacked Chromium; all checks passed after installation. - Reproduce the database measurements with `pnpm --filter @paperclipai/server exec tsx scripts/benchmark-sidebar-subtasks.ts`. - Extra local shard diagnosis ran 164 suites. Under this shell's `0077` creation mask, 157 suites passed and three had fixture permission or timing failures. Rerunning those three with `0022` resolved the permission failures. The remaining ten cases passed with one worker: two workspace cases and eight archive cases. The eight archive cases also passed on the pre-performance-fix revision, whose archive source is identical. The broad local shard is not claimed as one passing execution. The same-head CI retry supplies complete final CI coverage. - Nothing is merged or deployed. ## Risks - Exact counts still grow with the readable hierarchy and comment history. The change reduces repeated work; it does not cap or approximate counts. - A relevant summary batch can contain several recent parents. Older servers retain a broader summary fallback until upgraded. - Live delivery adds bounded, permission-checked parent lookups. Short delivery-window reuse limits repeated reads. - The authorization fast path applies only to open tasks with no privacy parent and no project. Restricted cases retain the shared predicate. - No migration is required. ## Model Used OpenAI Codex, based on GPT-6. Used code execution, native GitHub tools, PostgreSQL and Chromium verification. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either linked a public issue or described the issue in this PR - [x] I have not referenced internal or instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal issue id - [x] I have run focused tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation - [x] I have considered and documented risks - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open findings - [x] I will address all review comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
eb65fc0941 |
fix(server): turn agents' made-up-host Paperclip links into relative routes (#21762)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents post chat replies and task comments that link board pages and
attachments
> - Some models invent a hostname for these links, such as
`https://paperclip.local/COM/agents/<id>` or
`https://paperclip/api/attachments/<id>/content`
> - Those hosts do not exist, so the person clicks a broken link
> - The chat guidance already says never to invent a hostname, but a
model can still do it
> - This pull request rewrites such links to relative app routes before
an agent comment is saved, and adds the rule to the Paperclip skill
> - The benefit is that every board and attachment link an agent writes
opens, on whatever origin the reader uses
## Linked Issues or Issue Description
**What happened?**
Chat-eval cohort 16 (OpenCode with
`openrouter/deepseek/deepseek-v4.1-flash-1017`, master
|
||
|
|
348ed0ae10 |
fix(runner): keep closing tool-call tags and model control tokens out of OpenCode chat (#21761)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The Paperclip Runner OpenCode driver streams model text into the
chat as progress and selects the final reply
> - DeepSeek v4.1 flash (through OpenRouter) wrote tool-call scaffolding
and tokenizer control tokens as ordinary text
> - #21361 hides tool-call markup, but its detector matched only opening
DSML tags, and nothing recognised control tokens
> - A person saw raw `</|DSML| invoke>` lines in a progress message, and
in another conversation got a degenerate reply instead of an answer
> - This pull request extends the same detector to closing DSML tags and
to model control tokens, on the streaming path and at completion
> - The benefit is that scaffolding never reaches the chat, and a
degenerate reply goes to the existing correction instead of being posted
## Linked Issues or Issue Description
Refs #21361
**What happened?**
Chat-eval cohort 16 (OpenCode with
`openrouter/deepseek/deepseek-v4.1-flash-1017`, master
canary/v2026.1115.0-canary.13
|
||
|
|
56654adecd |
perf(server): reuse sandbox provider schemas for five seconds on the remote-run path (#21758)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agent runs can use remote sandboxes that a sandbox provider plugin supplies (for example Daytona) > - Each sandbox operation resolves the environment config, and that step reads the provider config schema to find the secret-ref fields > - On a hosted instance with 7 remote runs, this schema read was the most frequent statement: 17.95 per second, about 2.5 per second for each run > - The server runs all agents on one Node main thread, so each statement costs main-thread time; at 1,000 agents this read alone is about 2,500 statements per second > - This pull request reuses the retained schema for a short, bounded time and drops it at once when the plugin registry changes a plugin manifest > - The benefit is that a live remote run reads the schema about once per 5 seconds for each provider, not once per operation ## Linked Issues or Issue Description **What existing behavior does this improve?** Every plugin sandbox operation (exec, file sync in/out, runner ingress attach, capability snapshot, workspace realize, lease resume, lease release) calls `resolveEnvironmentDriverConfigForRuntime`. That call runs the `with selected_provider as materialized ...` statement in `resolvePluginSandboxProviderConfigSchema`, every time. **Subsystem affected** Server: plugin environment driver (`server/src/services/plugin-environment-driver.ts`) and the plugin registry manifest writes. **Current behavior** The schema read runs once per operation. A census test that drives the real runtime calls of one remote run counts 19 schema reads for 19 operations (see the table below). The hosted measurement was 17.95 executions per second with 7 runs. **Proposed behavior** The read runs at most once per 5 seconds for each provider and database client. A plugin install, upgrade or uninstall through the plugin registry in the same process takes effect on the next operation. **Reason and benefit** The schema is instance-wide plugin manifest data. It does not change during normal runs. Re-reading it per operation spends main-thread and database time for no new information. **Breaking changes** None. No API, schema or saved-state change. ## What Changed - `resolvePluginSandboxProviderConfigSchema` now returns the retained schema without a database read when the entry was checked less than `PROVIDER_SCHEMA_REUSE_MS` (5,000 ms) ago and the plugin manifest revision did not change. After that, it uses the existing cheap revision check, as before. - A lookup sequence number stops an older, slower lookup from replacing a newer retained entry. - A missing provider is still never retained, so a provider that becomes available is used on the next operation. - New module `plugin-manifest-revision.ts` holds an in-process manifest revision counter. It is a separate module so that the registry and the driver do not import each other. - `plugin-registry.ts`: `install` (both the new-row and the reinstall paths), `update` and `uninstall` announce a manifest change after the write completes. This is 5 changed lines plus 1 import line. Status-only writes do not announce, because the schema read does not use plugin status. - `plugin-loader.ts`: `installPlugin` runs `registry.install` inside an outer transaction. It now announces the manifest change again after that transaction settles, so a schema read made before the commit cannot be reused after it. This is 1 changed line plus 1 import line. - Statement census per operation, before and after. One remote run, one secret-ref field in the provider config, embedded test database, fake in-process plugin worker: | Operation | Count | Schema reads before | Schema reads after | All statements before | All statements after | | --- | --- | --- | --- | --- | --- | | Lease acquire | 1 | 1 | 1 | 26 | 26 | | Runner attach (ingress endpoint) | 1 | 2 | 0 | 11 | 9 | | Capability snapshot | 1 | 1 | 0 | 6 | 5 | | Exec | 10 | 10 | 0 | 50 | 40 | | File write (sync in) | 1 | 1 | 0 | 5 | 4 | | File read (sync out) | 1 | 1 | 0 | 5 | 4 | | Workspace realize | 1 | 1 | 0 | 5 | 4 | | Lease resume (reattach) | 1 | 1 | 0 | 6 | 5 | | Lease release | 1 | 1 | 0 | 18 | 17 | | **Total** | 18 | **19** | **1** | 132 | 114 | - Other reads that stay on each operation (reported, not changed in this pull request): each exec resolves each secret-ref field with 3 reads (secret, binding, version) and 1 `secret_access_events` insert. Runner attach and the capability snapshot also read the plugin row by id. These are part of the secret authorization and audit path, so they keep their own checks. ## Verification - `cd server && npx vitest run src/__tests__/plugin-sandbox-provider-schema-census.test.ts` (new). On master the census test fails with 19 schema reads. With this change it passes with 1. - The new tests also prove these points: - A registry upgrade that changes the secret-ref fields takes effect on the next call in the same process. - A direct manifest change (as from another server process) takes effect after the 5-second bound, and no read occurs inside the bound. - A registry uninstall takes effect at once. - A plugin reinstall through `pluginLoader.installPlugin` takes effect after its install transaction commits, even when a schema read ran inside the uncommitted transaction. This test failed before the post-commit announcement and passes with it. - A missing provider is not retained. The next call after install reads and uses it. - Two companies share the provider schema but each gets only its own secret values. A cross-company secret id still fails with "Secret must belong to same company". - A new database client (a process restart) starts cold, as before. - `cd server && npx vitest run src/__tests__/plugin-database.test.ts`: passed (plugin install transaction paths). - `cd server && npx vitest run src/__tests__/sandbox-provider-schema-reads.test.ts`: the existing tests now step the clock past the bound where they change the database directly. - `cd server && npx vitest run src/__tests__/environment-runtime.test.ts`: 210 passed. - `cd server && npx vitest run src/__tests__/bundled-plugin-registration.test.ts src/__tests__/external-object-refresh-work.test.ts src/__tests__/plugin-lifecycle-restart.test.ts src/__tests__/plugin-loader-error-retry.test.ts src/__tests__/plugin-environment-driver-sandbox-capabilities.test.ts src/__tests__/plugin-sandbox-provider-config-validation.test.ts src/__tests__/environment-probe.test.ts`: 82 passed, 1 skipped. - `cd server && pnpm typecheck`: passed. ## Risks - Staleness bound: a plugin manifest change made by a different server process (for example during an overlapping deploy) can take up to 5 seconds to apply here. During that time, a secret-ref field that an upgrade adds is not resolved yet, and a field that an upgrade removes is still resolved. The bound is short and it applies only to which config paths are secret refs. - The retained value is instance-wide plugin manifest data, keyed by database client and driver key. It contains no company data and no secret values. It does not take part in an authorization decision. Secret resolution, the environment binding check and the access audit still run on each operation for the calling company. - Memory stays bounded by the existing limits: 32 entries per database client, 64 KiB per schema. - `plugin-registry.ts` has a small edit in `install`, `update` and `uninstall`. Other open work in that file can cause a simple merge conflict. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edit, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Refs #17581 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
15afc32137 |
test(scaling): wait for agent setup and gate the ramp on a first started run (#21757)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The scaling effort uses a real-agent ramp harness (`scripts/scale-agents-ramp.ts`, added in #21695) to load a hosted test instance through the product API > - Its first real run held a 20-minute step with 10 tasks, 10 queued runs and 0 started runs, and reported only "concurrency_reached failed (observed 0)" > - The cause was in the harness: it bound the agents to a plugin Runner-task environment, which runs only agents with a managed AI connection. Each agent stayed in setup (status `paused`, setup state `verifying`) with that reason, so each run stayed queued. The product did what it is designed to do > - The harness did not wait for agent setup, did not read the setup error, and held the full step with nothing running. Its fake API created agents ready at once, so its tests could not catch this > - This pull request makes the harness wait for agent setup, gate the ramp on one started run, and stop within minutes with Paperclip's own reason > - The benefit is that a wrong environment or any other start blocker costs about one minute and prints the cause, not a 20-minute step with no explanation ## Linked Issues or Issue Description Refs #21695 **What happened?** A ramp of 10 then 50 agents on a hosted test stack created 10 agents, 10 tasks and 10 runs. In 20 minutes no run started. The result said only "concurrency_reached failed (observed 0)". Cleanup found every agent already paused and cancelled 10 queued runs. The tasks never left `todo` (status version 1, no checkout run). The agents were bound to the "Paperclip Sandbox (Daytona)" plugin environment, whose `daytona-tasks` driver declares `supportsTasks: true`. For such an environment, lifecycle verification refuses a `paperclip_runner` agent that has no `runtimeConfig.aiConnection` (`ai_connection_required`, see `server/src/services/agent-harness-verification.ts` and its test "refuses a plugin Runner-task agent without a managed AI connection before allocating a target"). The failure pauses automatic verification, so the agents never become ready. **Expected behavior** The harness must not create a full step of work before agents can run. When agents cannot finish setup, or when no run starts, it must stop soon, print the task, run and agent state with Paperclip's reason, and clean up. **Steps to reproduce** 1. Run `node --test scripts/scale-ramp/scale-ramp.test.ts` with the new fake (agents start in setup; `agentSetupError` keeps them there) and the old `ramp.ts`, `setup.ts` and `config.ts`. 2. The new test "stops within minutes with Paperclip's reason when the agents never finish setup" fails with `concurrency_reached`, the same result as the real run. 3. With this change, it stops with `agents_ready` and the setup reason in under 5 virtual minutes. **Paperclip version or commit** Instance on `2026.1106.0+2555.git.819f61f1f` (master at 11:34Z on 2026-11-15); harness from #21695. ## What Changed - `setup.ts`: add `waitForAgentsReady`, `agentReady` and `describeAgent`. The harness reads the company agent list until each agent has setup state `ready` and is not paused. It stops when an agent reports a setup error for 60 s, or after `--agent-ready-minutes` (default 10), and prints the agent's status, setup state, holds and Paperclip's reason. - `ramp.ts`: before the first step, a preflight creates one agent and one task and creates the rest only after that task's run starts. A new check stops the ramp when no run has started within `--start-timeout-minutes` (default 5) of the first task. The stop reason names the task status, any execution blocker, the run status, error code, error and liveness, and the agent's setup state. New stop criteria: `agents_ready` and `run_started`. Run records keep `error`, `errorCode` and `livenessReason`. - `config.ts`: add `--start-timeout-minutes` and `--agent-ready-minutes`. - `fake-paperclip.ts`: agents start as the real product creates them (status `paused`, setup state `preparing`) and become `ready` after `agentReadyDelayMs`, or stay in `verifying` with `agentSetupError`. Issue creation queues a run even for an agent in setup, as the real product does. Runs start only for a ready agent. - `scale-ramp.test.ts`: three tests. Agents that never finish setup stop the run with `agents_ready` and the reason, with one agent and no task created. A first task that never starts stops the run at the preflight with `run_started`, the task, run and agent state, and one cancelled run. The rest of a step is created only after the preflight task starts. - `doc/plans/2026-11-13-thousand-concurrent-agents.md`: the Phase 3 runbook describes the start checks, the first run's cause, and which environments the harness can use. ## Verification - `node --test scripts/scale-ramp/scale-ramp.test.ts`: 43 of 43 pass. - Red check: with the new fake and tests but the `origin/master` `ramp.ts`, `setup.ts` and `config.ts`, 3 tests fail. The setup test fails with `concurrency_reached` (the real run's result) where `agents_ready` is expected. - `tsc -p scripts/scale-ramp/tsconfig.json` (same options): no errors. - Read-only inspection of the hosted instance (GET only): task rows (`todo`, status version 1, no checkout run), activity (agent created, no pause, 10 `heartbeat.cancelled` at cleanup), the plugin manifest (`daytona-tasks` `supportsTasks: true`), and the company AI connections (no OpenRouter connection). The run and agent rows were deleted with the agents at cleanup, so their setup error was inferred from the code path and the cleanup counts, not read. - Not run: a new real ramp (needs the owner's go-ahead). ## Risks - Low risk. The change touches only the scale-ramp harness, its fake and its tests, and one runbook section. No product code changes. - A real ramp now spends up to one preflight start (seconds to minutes) before the first step. The preflight task is part of step 1 and the preflight running samples are dropped, so step metrics keep their meaning. - An agent whose setup error clears within 60 s does not stop the run. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, read-only HTTP GETs to the test instance). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
3244780bd0 |
fix(queue): keep a message a starting run's first turn took out of the queue (#21747)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A message posted while a run is starting goes into that run's first turn, and the task page shows it as sent > - The page shows it as sent only while the live queue lists it as a first-turn message > - When that queue closes, for example after a later Steer, the page has no lasting record of the delivery > - It then infers that the message is queued and shows it again with a Steer button > - This pull request reports each run's own first-turn receipt in the run list, and the page places those messages in that run > - The benefit is that a sent message stays sent ## Linked Issues or Issue Description Follow-up to #21661, found while I verified that change on staging. **What happened?** With a Codex runner, two messages were posted while the run was starting. The first turn took them and the agent acted on both. Then a third message was steered into the running turn. After the Steer, the first two messages came back into the queue with Steer buttons. The server queue was empty. **Expected behavior** Messages that a run's first turn took stay in the conversation as delivered. **Steps to reproduce** 1. Start a task with a Paperclip runner agent. 2. While the runner is starting, post one or two messages. 3. After the turn starts, post another message and click Steer. ## What Changed - `activityService.runsForIssue` returns `firstTurnCommentIds` from the run's `firstTurnQueuedInput` receipt. The field is set only when the receipt names this run, because a retry copies its predecessor's context. - `IssueDetail` adds those ids to the run's start inputs, so the messages are placed in that run and are not inferred as queued. - `RunForIssue` type gets the new optional field. ## Verification Red/green tests. Each failed before the fix and passes after it: - `server/src/__tests__/activity-service.test.ts`: "reports the queued messages a run's own first turn took, not a predecessor's". 33/33 pass. - `ui/src/pages/IssueDetail.test.tsx`: "keeps a message the running turn already took out of the queue after the live queue closes". The IssueDetail, queue helper, and task-chat suites pass (69 files, 1298 tests). UI typecheck passes. Staging, after #21661 deployed, Codex runner: two messages posted while the run started were taken into the first turn and acted on. No Interrupt, error banner, or composer wait was shown. A mid-turn Steer was accepted in seconds. This PR fixes the one defect that the check found. ## Risks - Low. One new read-only field on the run list. A run that took messages and then failed before its provider accepted the turn now shows those messages in that run. The next run still delivers them, and the queue still lists them while they wait. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>canary/v2026.1115.0-canary.12 |
||
|
|
e7f31705da |
perf(release): stage the verified host release daemon instead of compiling it in publish_canary (#21555)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Every master push runs a canary, and only one canary runs at a time, so its duration sets how often merged work ships > - `publish_canary` takes about 30 min. 8 min of it is `pnpm build`, and 4 min 40 s of that is an uncached `cargo build --release` of `paperclip-runnerd` > - The release then replaces that binary with the exact-source daemons that `release_runner_binaries` already built and verified for all four targets > - This pull request stages the verified daemon for the publisher's own target during a release, instead of compiling it > - The benefit is about 4.5 min less per canary, with the same verified bytes published ## Linked Issues or Issue Description **What happened?** In canary run 43487884481 (2026-11-13), the `Publish canary` step took 28.6 min. Its `pnpm build` ran 19:42-19:50. The `packages/paperclip-runner build` took 5.3 min, and its `cargo build` reported `Finished release profile ... in 4m 40s`. Every other package waited for it. `release.sh` then runs `assemble-daemons`, which overwrites `dist/bin` with the four exact-source release daemons. **Expected behavior** A release does not compile a binary that it discards. The Runner build stamp stays valid, so the server build still reuses the Runner build. Refs #21554 (the other canary speed-up) ## What Changed - `packages/paperclip-runner/scripts/build-runner-binary.mjs` (new): when `PAPERCLIP_RELEASE_RUNNER_ARTIFACTS_DIR` is set, it checks the release daemons with the existing `releaseDaemonManifest` (source revision equal to `HEAD`, digest, architecture, build metadata) and stages the host target's daemon at `dist/bin` with the existing `stageRunnerBinary`. Otherwise it runs `pnpm run build:binary` as before. - The Runner `build` chain calls this script in place of `pnpm run build:binary`. `build:binary` itself and the image build are unchanged. - `stage-release-runner-binaries.mjs` exports `metadataContract`. - Tests: `stage-runner-binary.test.mjs` (staging per target, wrong source, bad digest, missing target, non-release fallback); `runner-image-identity.test.ts` step list. - `doc/RELEASING.md` describes the behavior. ## Verification - `node --test packages/paperclip-runner/scripts/stage-runner-binary.test.mjs`: 9 of 9 pass. - `server`: `vitest run src/__tests__/runner-image-identity.test.ts`: 42 of 42 pass. - With the real daemon artifacts of canary run 43487884481 (receipts pointed at this commit): `PAPERCLIP_RELEASE_RUNNER_ARTIFACTS_DIR=... pnpm run build` in `packages/paperclip-runner` finished in 15 s with no `cargo` line. The build stamp was recorded, and `server/scripts/prepare-runner-vendor.mjs` then reported `action:reuse,reason:inputs_unchanged`. The staged daemon runs. - After merge: the next canary's `packages/paperclip-runner build` log shows "Staged the verified exact-source linux-x64 release daemon" and no `Compiling` lines. ## Risks - Low. A release still fails closed: a daemon with the wrong source, digest, architecture or metadata stops the build, and `assemble-daemons` checks all four again afterwards. - Local, Docker and CI builds without the variable are unchanged. - On macOS, staging re-signs the binary, as it does for a compiled one. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>canary/v2026.1115.0-canary.11 |
||
|
|
8cd2d6a73d |
docs(skills): add aggregate-bound, lost-runner, and bug-class lessons to fix-failed-paperclip-run (#21745)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The `fix-failed-paperclip-run` skill guides agents that repair failed or stuck runs > - A recent repair series made automatic recovery more aggressive, and three lessons came out of it > - The aggregate number of automatic run starts was not bounded, so an outage produced many failing starts > - CI jobs lost to runner shutdowns and tests broken on master held merges > - Review rounds kept finding new instances of one bug class, one at a time > - This pull request adds those three lessons to the skill in eight lines > - The benefit is fewer repair rounds and safer recovery changes ## Linked Issues or Issue Description **What happened?** Recovery repairs bounded attempts per task but not the total number of automatic run starts across a company. During an outage that total was large and every start failed. CI jobs lost to runner shutdowns held merges. Review found one instance of a bug class per round. **Expected behavior** The skill tells agents to state the aggregate bound for recovery changes, to re-run lost CI jobs, and to check a whole bug class after one instance. **Steps to reproduce** 1. Read `.agents/skills/fix-failed-paperclip-run/SKILL.md` on master. 2. Observe that none of the three points is covered. Refs #21443, #21561, #21394. ## What Changed - Section 4: a repair that makes recovery start more runs states its worst-case company-wide start rate during an outage and what bounds it. - Section 6: re-run CI jobs lost to a runner shutdown; re-run an unrelated failing test once, then check master and tell its owner; after one instance of a bug class, check every entry point or counter of that class. ## Verification - `git diff --check`: no errors. - Documentation only; no code or tests changed. ## Risks Low risk. The change adds eight lines of guidance to one skill. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`), in Claude Code, with tool use. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
c17ef9249d |
fix(comments): keep saved authorization cards from blocking replies (#21744)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - People use task comments to ask for help with pending requests. > - Saving a comment also checks whether it replaces any pending requests. > - A stored request that fails today's payload schema can abort that comment write. > - The connection gateway could write such a request without validating its generated target. > - This change validates new authorization cards, reads older custom identities, and keeps malformed requests from blocking comments. > - Comments remain available while authorization and review gates retain their authority. ## Linked Issues or Issue Description **What happened?** A connection authorization card can contain a generated custom target key longer than 120 characters. The gateway writes the card directly. Later, the interaction reader rejects it. Both the task interaction list and comment submission then return a validation error. Other malformed saved requests can also abort a comment write. **Expected behavior** People can ask questions on a task while authorization is pending. New cards meet the shared schema. Older generated identities remain readable. A malformed request never becomes approval because a person posted a comment. **Steps to reproduce** 1. Use a personal connection with a long UID and no grant for the current user. 2. Invoke a tool. The gateway saves an authorization request with the long UID in its target key. 3. Open the task or post a comment. Reading the saved payload fails the key length limit. Related PR searches found no existing repair for this defect. This is a focused bug fix, with no roadmap feature overlap. ## What Changed - Validate generated connection authorization payloads before either insert or update. Use a stable SHA-256 target identity from the existing connection/user idempotency key. Bound its display label. - Read older confirmation custom keys and labels without the new-input length limits. Preserve their exact identities. Keep target types, URLs, document revisions, and resolver checks strict. New requests still use the strict shared schema. - During current and historical comment supersession, keep an invalid saved request pending, report its ID and schema locations, and continue with valid siblings. Database and other unexpected errors still propagate. - Keep this behavior in the gateway writer and interaction owner. The two existing source files exceed 1,000 lines; the additions extend their existing write/hydration boundaries rather than introduce another ownership path. ## Verification - Reproduced the deployed failure through the browser: comment POST and interaction GET both returned HTTP 400 for `target.key` exceeding 120 characters. The person's draft was restored. - Ran the new regressions before the repair: four expected failures and one short-identity control pass. Failures came from real payload parsing and real PostgreSQL comment writes, not fixture setup. - After repair, all 257 tests passed in the complete affected suites: `issue-thread-interactions-service.test.ts`, `tool-gateway.test.ts`, and `issue-thread-interactions.test.ts`. - Safety cases cover fresh and reused authorization cards, repeated missing-grant calls, refusing another user's credentials, two persisted follow-ups, strict new-input limits, wrong-company resolution, unsafe URLs, invalid document revisions, malformed question and confirmation payloads, and valid sibling supersession. - Local browser acceptance used an isolated, credential-free source instance on macOS arm64 with embedded PostgreSQL and Chrome. A card shaped like an older saved long-key authorization request remained visible. Two comments saved through Send, including a second after reload. The original comment persisted and the authorization request remained pending. No model execution was configured. - Browser acceptance was repeated on final commit `58b57dfe104b2ab421c7f24c89db4c29fcfda8ec` after rebase. The server health and ownership check confirmed that exact serving revision. Both comments persisted and the authorization stayed pending. Both owned instances were then stopped. All 257 affected tests passed again after rebase. The final fixture typing correction passed its six affected PostgreSQL cases. Greptile reviewed final head `58b57dfe104b2ab421c7f24c89db4c29fcfda8ec` at 5/5, with zero review threads or open findings. Required CI is green on that head: aggregate `ci / verify` and `ci / e2e` both passed. The repository CI run is https://github.com/paperclipai/paperclip/actions/runs/43710675824 . All 55 completed checks passed; four inapplicable checks were skipped. The non-required Superagent Security Scan remains in progress at 35%, with no annotations or reported findings. This external scan is still unverified; it was not retried or marked as passed. - `git diff --check` and the module boundary check pass. All seven local server typecheck layers pass on the final fixture revision. An earlier run found fixture typing errors; those were corrected and rerun successfully. Full repo tests, release qualification, and deployment were not run. ## Risks - Stored custom keys and labels may exceed current input limits. Only those length limits change for saved confirmation payloads; unsafe links and invalid document targets still fail validation. - A malformed request stays pending and still needs repair. It does not receive consent, disappear, or block a comment. Its read or resolution can still return a validation error. - This PR does not deploy the fix or recover a remote task. The original environment remains affected until rollout. ## Model Used OpenAI GPT-6-based Codex, with code editing, shell execution, browser tools, and PostgreSQL-backed tests. The host does not expose a more specific model version or context size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes (behavior and limits are recorded in this PR) - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
689d62661d |
fix(server): wake a delivery task unscoped when it cannot apply settled hints (#21686)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The delivery work coordinator wakes background tasks when a database work topic settles, and passes the settled scope hints to the task's `onSettled` > - The coordinator calls `onSettled(hints)` before `wake()` > - If `onSettled` throws on a hint that it does not recognise, the task is not woken. Only a generic listener warning is logged > - Hints now cross server processes (#21681), so a hint shape from another server version can arrive during a deploy overlap > - This pull request makes the coordinator log a warning, apply the settlement unscoped once, and always wake the task > - The benefit is that a hint that a task cannot read never makes the task miss committed work ## Linked Issues or Issue Description Follow-up to #21681 (merged). This branch is based on master. **What existing behavior does this improve?** How the delivery work coordinator handles an `onSettled` callback that throws. **Subsystem affected** `server/src/services/delivery-work-coordinator.ts`. **Current behavior** `subscribeDatabaseWork` calls the coordinator listener, which calls `task.onSettled?.(hints)` and then `wake()`. If `onSettled` throws, `wake()` does not run. The work-signal publisher catches the error and logs "Database work listener failed". The task stays asleep until a different event or deadline wakes it. **Proposed behavior** If `onSettled(hints)` throws, the coordinator logs a warning and calls `onSettled(undefined)` once, which is the unscoped settlement. If that also throws, it logs a second warning. In all cases it then wakes the task. **Reason and benefit** A settled event means committed work can be ready. A task must never miss it because a hint could not be applied. **Breaking changes** None. ## What Changed - `delivery-work-coordinator.ts`: a settled event goes through `applySettled(hints)`. It tries `onSettled(hints)`. If that throws and hints were given, it tries `onSettled(undefined)` one time. Each failure logs `Delivery settlement could not be applied for <queue>` with `process.emitWarning`. Then `wake()` runs. - Unscoped means `onSettled(undefined)`. Each registered task already defines what that means. For example, scoped deadline queues, dispatch work, muse maintenance and the revocation index mark `*` dirty. Counter-style tasks set their request flag. `chat-action-work` does nothing. The fallback is bounded: it calls `onSettled` at most twice per event, it does not recurse, and it does not schedule another settlement. So it cannot loop. - `work-signals.test.ts`: a new test pins one rule of #21681 for the local path and the remote path. An unhinted `signalDatabaseWork` in the same transaction as a hinted one gives a settlement that is scoped to the hinted set. The owner of `work-signals.ts` confirmed that this is deliberate. The repository does not allow code comments in pull requests, so this sentence and the test document it. ## Verification - `cd server && npx vitest run src/__tests__/delivery-work-coordinator.test.ts`: 31 pass. New tests: - An `onSettled` that throws on an unknown hint shape: the task gets `[hint]` and then the unscoped settlement, it is woken once, and one warning is logged. `onError` is not called. - An `onSettled` that always throws: the task gets 2 calls and is woken once, with 2 warnings. A later unscoped event gives 1 call, 1 wake and 1 warning. No more runs happen after 60 seconds, so there is no loop. - A normal `onSettled`: it gets the hints once, the task is woken once, and no warning is logged. - Red/green: with the coordinator from master, the 2 throwing tests fail. `onSettled` is called only with the hints, and there is no unscoped settlement and no wake. With this branch, all tests pass. - `cd packages/db && npx vitest run src/work-signals.test.ts -t alongside`: passes on real PostgreSQL. The local listener and the receiving process both get `["company-1"]`. ## Risks - Low risk. The change applies only when `onSettled` throws. Before, that path skipped the wake. Now it wakes with an unscoped settlement. For scoped deadline queues an unscoped settlement is a `*` reload. That is the existing safe fallback, and the reconstruction interval of the queue still throttles it. - A task whose `onSettled` throws on every event now runs on every settled event and logs a warning each time. Before, it did not run. The warning makes such a defect visible. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edit, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
fb3b09aaef |
refactor: split the runner-e2e and restart-chaos READMEs into a runbook and a catalog (#21743)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - `tests/runner-e2e` is the paid Product E2E harness and `tests/restart-chaos` is the restart chaos harness > - A README is the first file a person or an agent opens in a directory > - These two READMEs grew to 3,805 and 2,277 lines. Most of each is a catalog with one section per suite or per fault scenario, and almost none of it is ever trimmed > - This pull request splits each README into a runbook and a catalog file > - The benefit is a README that can be read in full, and a clear place for the next suite or scenario note ## Linked Issues or Issue Description Refs #21670 and #21700 (the same cleanup for `doc/DEVELOPING.md` and `doc/DATABASE.md`). **Issue type** Outdated or confusing documentation. **Where is the issue?** `tests/runner-e2e/README.md` and `tests/restart-chaos/README.md` **What's wrong?** In the last 60 days, `tests/runner-e2e/README.md` gained 3,476 lines and lost 182. `tests/restart-chaos/README.md` gained 2,411 and lost 134. The setup, evidence, and CI sections sit between suite notes. The first 225 lines of the runner-e2e README are suite notes with no heading. **Suggested fix** Keep how to run the harness in the README. Move the per-suite and per-scenario sections to a catalog file in the same directory. ## What Changed - `tests/runner-e2e/README.md` is now 767 lines: credentials, local commands, the Daytona image, evidence and cleanup, billing, the dashboard, GitHub Actions, and the report preview. - New `tests/runner-e2e/SUITES.md` (3,055 lines) holds every per-suite section. The two blocks of suite notes that had no heading now have one. - `tests/restart-chaos/README.md` is now 592 lines: run, fault timing, Daytona mode, hosted mode, universal checks, grading notes, pieces, output, and the crash-point suite. - New `tests/restart-chaos/SCENARIOS.md` (1,694 lines) holds every workload and fault scenario section. - All text is moved without wording changes. Nothing is deleted. The only edits are three "(below)" references in the restart-chaos README, which are now links. - Repointed 22 anchor links and 3 plain links in `doc/evals.md` and `maps/feature-map/cases.md`. - Updated seven code comments in `tests/restart-chaos/` that name a moved section, and two skills that list the runner-e2e docs to read. ## Verification - The split script asserts that each source line is kept or moved exactly one time, and checks the first line of each range. - A link check on the four files and on every file that links into them found 0 missing files and 0 missing anchors. - `tests/runner-e2e/daytona-image.test.ts` reads the README for a Docker build recipe. The recipe is still in the README. I did not run that test locally. - `node scripts/check-maps.mjs` passes. - `node --test .github/scripts/tests/*.test.mjs`: 808 pass, 0 fail. - `pnpm verify` was not run. This worktree has no installed dependencies. ## Risks - Low risk. The change is documentation and comments only. - A link to a moved section from outside this repository lands on the top of the README. - The catalog files are still long. This change moves them off the first-read path. It does not prune them. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`), Claude Code desktop, with tool use. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [ ] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
066c8244b5 |
perf(release): build canary daemons during source verification and keep their dependency cache (#21554)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Every master push runs a canary that publishes npm packages and reaches staging > - Only one canary runs at a time, so a canary's duration sets how often merged work ships > - A canary takes about 70 min. Two parts of it are avoidable waiting: the Runner daemons build only after the source proof, and darwin-x64 often rebuilds every dependency crate > - This pull request starts the daemon builds with the canary and keeps their dependency cache across Runner changes > - The benefit is about 15-25 min less per canary, with no change to what must pass before publication ## Linked Issues or Issue Description **What happened?** In successful canaries on 2026-11-13 the critical path was: `Reuse exact-source verification` 11-14 min, then `Release daemon (darwin-x64)` 10-28 min (9 runs: 10, 16, 18, 20, 20, 21, 24, 24, 28), then `publish_canary` 27-37 min. The daemon builds waited for the source proof, although they only build and upload artifacts. The darwin-x64 build was 16.4 min with a cache miss and 3.9 min with a hit. The native cache key included a hash of `packages/paperclip-runner/src/**`, `protocol/**` and `*.json`, but the cache holds dependency crates only (`cache-workspace-crates: false`). So every Runner change caused a full dependency rebuild. **Expected behavior** Daemons build in parallel with the source wait. The dependency cache changes only when dependencies, Cargo configuration, host or target change. Refs #21399, #16028 ## What Changed - `release.yml`: `release_runner_source` no longer needs `verify_canary`. On a canonical master push it starts at once. `publish_canary` still requires `verify_canary.result == 'success'`, so nothing publishes from unverified source. - `release-verify.yml`: the native Rust cache `shared-key` drops the Runner source hash. It keeps the Cargo manifest and configuration, host and target. - `release-verify-workflow.test.mjs`: tests that a canonical master push selects the source without the proof, that a fork, other ref or other event does not, that `publish_canary` still needs the proof, and that the cache key contains no source globs. - `doc/RELEASING.md`: describes both behaviors. ## Verification - `node --test scripts/__tests__/release-verify-workflow.test.mjs`: 73 of 73 pass. - Red: the same tests against master's `release.yml` and `release-verify.yml` fail 2 tests (the source selection and the cache key). - After merge: the next canary shows `Release daemon (*)` starting at the same time as `Reuse exact-source verification`, and darwin-x64 restores its cache. ## Risks - Low. If verification fails, that commit's daemon builds are wasted macOS minutes. Verification failures on master are now rare. - The first canary after merge saves a new cache entry, so it is not faster for darwin-x64. Later canaries are. - Stable, beta and nightly still select their source through their own gates, unchanged. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
83541e89ad |
perf(recovery): re-check only changed tasks between hourly full backstop passes (#21729)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The server runs issue recovery queues on one Node main thread. `stranded-assignments` and `assigned-work-invariant` find assigned tasks that have no live run and restart them. > - #21552 scoped run hints to the run's task graph and kept a company "backstop" pass to catch a missed hint. Any graph hint arms the backstop, and the backstop runs at most every 5 minutes after the last full pass. > - The backstop reloads and re-checks every open assigned task of the company. With any trickle of activity it runs every 5 minutes. On the hosted staging instance that is about 850 tasks every 5 minutes, about 2.8 checks per second at idle, close to the measured 3.2 per second. That is work proportional to history, not to activity. > - This pull request makes the 5-minute backstop re-check only tasks whose own rows changed since the previous look, with one set-based query. A full company pass runs at most once an hour, still armed only by activity. > - The benefit is no re-checks between hourly passes when nothing changed: idle statements drop from 59 per second to 5.5 per second in a staging-shaped fixture. A task whose own state changed without a hint is still recovered within 5 minutes. ## Linked Issues or Issue Description No issue exists. This describes the problem with the fields of `.github/ISSUE_TEMPLATE/enhancement.yml`. **What existing behavior does this improve?** The missed-hint backstop of the `stranded-assignments` and `assigned-work-invariant` recovery queues (`server/src/services/recovery/work.ts`). **Subsystem affected** Server: issue recovery background work. **Current behavior** Each graph hint arms a `{companyId, backstop: true}` scope. When it opens (5 minutes after the last full pass), the queue loads every open assigned task of the company and checks each one again: about 22 statements per task for the stranded check. With a hint at least every 5 minutes, this repeats every 5 minutes, whatever changed. **Proposed behavior** Between full passes, the backstop loads only the tasks whose own rows changed since the previous look (minus a 5-minute overlap for late commits): the task row, its runs, its recovery actions, its wake requests, its thread interactions, its comments, or its assignee agent. A full company pass runs when the last one is an hour old. A real change still sends its own hint and gets an immediate targeted check. **Reason and benefit** On staging at idle, the stranded check runs about 3.2 times per second with zero runs. A fixture with staging's population shape attributes all of it to the backstop. ## What Changed - `server/src/services/recovery/work.ts`: - `fullPasses()` keeps the time of the last full pass and of the last changed-tasks pass per company. The backstop gate is 5 minutes after the later of the two. - A backstop scan decides on its first page: a full load when the last full pass is at least `ISSUE_RECOVERY_FULL_BACKSTOP_MS` (1 hour) old, otherwise a changed-tasks load since the previous look minus `ISSUE_RECOVERY_BACKSTOP_MS`. All pages of one scan use the same decision. - Each scan records its start time on its first page. A completed scan advances the saved full or changed time only to that start, so a change behind the cursor of a slow scan falls in the next window. - The stranded queue adds `changedTaskCondition(since)` to its load query. The assigned-work queue passes `changedSince` to `listSuspects`. Both return `keepUnseen: true` for a changed-tasks load. - A new `fullBackstopMs` option. When a caller sets only `backstopMs` (tests), every backstop stays a full pass, as before. - With `PAPERCLIP_DEBUG_WORK_SIGNALS=stranded-assignments`, each `check` event now names why the task was loaded: `deadline`, `reload` (unscoped reconstruction), `backstop`, `company` (company-wide hint) or `hint` (task or graph hint). - `server/src/services/recovery/changed-tasks.ts` (new): the set-based "changed since" condition. Each part uses an existing index: `issues (company_id, updated_at)`, runs by `(company_id, context issueId)`, wakes by `(company_id, payload issueId)`, interactions and comments by `(company_id, issue_id)`. - `server/src/services/scoped-deadline-work.ts`: a load result may set `keepUnseen`. Then a completed scan does not drop the pending entries it did not load, so their saved deadlines stay. - `server/src/services/recovery/assigned-work-sweep.ts`: `listSuspects` accepts `changedSince`. - `server/src/services/issue-recovery-notifications.ts`: `ISSUE_RECOVERY_FULL_BACKSTOP_MS = 60 * 60_000`. - Tests: - `server/src/__tests__/issue-recovery-idle-backstop.test.ts` (new): the staging-shaped fixture, see Verification. - The staging-shaped population helper and the fixture's statement meter come from #21724, now merged into this branch. - `server/src/__tests__/issue-recovery-work-debug-attribution.test.ts`: the check event now names its load reason (`hint` or, when the backstop loaded the task last, `backstop`) instead of `load`. - `server/src/__tests__/issue-recovery-deadlines.test.ts`: the backlog helper writes its rows with the test's fake clock, as production writes use the server clock. Without it, rows stamped with the database clock look older than the fake-clock window. ## Verification - Fixture (`issue-recovery-idle-backstop.test.ts`, embedded PostgreSQL, the real recovery, dispatch and retry-cancellation workers, fake clock, statement meter, `PAPERCLIP_DEBUG_WORK_SIGNALS=stranded-assignments:1`): one company, 50 agents, 780 in-review tasks with succeeded native runs and no execution state, 40 in-progress and 30 to-do tasks of a paused agent, 12 tasks with an active agent-owned recovery action, 3 tasks of a terminated agent. After boot, one unrelated graph hint per minute stands in for idle activity. - Wake reasons on master over 30 idle minutes with that trickle: 5,100 stranded checks, all `backstop`; 0 from hints, 0 from company hints, 0 from deadlines. That is 2.83 checks per second and 106,632 statements (59.2 per second). With no hint at all after boot, master still made one backstop pass (850 checks) in 30 minutes, armed by boot activity. - This branch, same fixture: | | master | this branch | | --- | --- | --- | | Stranded checks, first 30 idle minutes | 5,100 | 0 | | Statements, first 30 idle minutes | 106,632 | 1,044 | | Stranded checks, one idle hour (one full pass) | about 10,200 | 850 | | Statements per second over that hour | 59.2 | 5.5 | - Missed hint: a task inserted with no hint is checked and gets a run within 5 minutes plus one pass (`ISSUE_RECOVERY_BACKSTOP_MS + 2 min` asserted). - A change outside the task's own rows (its agent resumed by a raw update that sets no `updated_at`, with no hint) is not seen by the 5-minute pass. It is checked and gets a run after more than 10 minutes and within 62 minutes, through the hourly full pass. - Prompt pickup with the backstop closed: a finished run (`notifyNativeStatusReassessment`), a user comment, a hold release (`pause_lifted`, which a budget raise and an agent resume send) and an agent resume each get a check of the affected task within 60 seconds. The company-wide events re-check the whole company at 25 tasks per pass, as before. - Two regression tests in `issue-recovery-deadlines.test.ts`, red on |
||
|
|
07d5a6ee17 |
perf(db): carry scope hints in cross-process work notifications (#21681)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Server processes share durable queues and wake each other with a
commit-bound PostgreSQL NOTIFY
> - The NOTIFY payload carries only `{version, origin, topic}`, so the
receiver gets an unscoped "settled" event
> - In `scoped-deadline-work.ts` an unscoped settle becomes `*`, which
reloads every candidate of that queue on the whole instance
> - On a large instance one cross-process notification can re-check
about 850 open tasks, at about 22 statements each
> - This blocks more than one server process per stack, and it costs
work during every deploy overlap
> - This pull request puts the topic's hints in the NOTIFY payload, so
the receiver reloads only the hinted scopes
> - The benefit is that a scoped commit in one process costs the same
scoped reload in every other process
## Linked Issues or Issue Description
**What existing behavior does this improve?**
Cross-process work notifications between server processes on one
database.
**Subsystem affected**
`packages/db` work signals (`work-signals.ts`) and their consumers, such
as `server/src/services/scoped-deadline-work.ts`.
**Current behavior**
`receiveDatabaseWorkNotification` publishes "settled" with no hints. A
scoped deadline queue then reloads all of its candidates, throttled only
by its reconstruction interval.
**Proposed behavior**
The NOTIFY payload carries the deduplicated hints of its topic. The
receiving process publishes "settled" with the same hints that a local
commit publishes. Overflow and malformed input fall back to the old
unscoped behavior.
**Reason and benefit**
A scoped commit no longer causes an instance-wide reload in other
processes. This is necessary before a stack runs more than one server
process.
**Breaking changes**
None. Old and new processes interoperate in both directions (see Risks).
## What Changed
- `packages/db/src/work-signals.ts`: the per-level pending notifications
are now a map of topic to `{ hints, unscoped }`. Savepoint merge and
rollback keep the same rules as before.
- The hints follow the local "settled" semantics: the union of the hints
from `signalDatabaseWork` and `publishDatabaseWorkSettled`. An unhinted
`publishDatabaseWorkSettled`, `notifyDatabaseWorkOnCommit`, or a topic
with no hints sends the unscoped payload.
- A hinted payload is `{version: 1, origin, topic, hints: [...]}`. The
field is additive and the version stays 1.
- Each payload is at most 1,024 bytes
(`DATABASE_WORK_NOTIFICATION_MAX_BYTES`), because current receivers drop
any payload longer than 1,024 characters before they parse it. Many
hints split into several payloads. The payloads of one topic are at most
7,000 bytes in total (`DATABASE_WORK_TOPIC_NOTIFICATIONS_MAX_BYTES`). If
a topic exceeds this cap, or one hint alone does not fit, the topic
sends one unscoped payload.
- All payloads still go in the one batched `select pg_notify(...) from
jsonb_array_elements_text(...)` statement at commit. There are no added
statements or awaits on the commit path.
- `receiveDatabaseWorkNotification` accepts payloads up to 8,000
characters (the PostgreSQL limit). It keeps the drop of own-origin
payloads and of payloads without a usable origin or topic. A missing,
malformed, or empty hint list, or a version other than 1, gives an
unscoped wake. It never throws.
- The `remote_notification` debug event now logs the received hints
(`hints`: a JSON array, or `null` when unscoped).
- Tests: new payload, compatibility and fallback tests in
`packages/db/src/work-signals.test.ts`, and a new two-process test on
real PostgreSQL in
`server/src/__tests__/scoped-deadline-work-notifications.test.ts`.
## Verification
- `cd server && npx vitest run
src/__tests__/scoped-deadline-work-notifications.test.ts`: two pools on
one embedded PostgreSQL database. Process A commits work. Process B runs
a real delivery coordinator with `registerScopedDeadlineWork` over 800
candidate rows in 20 companies.
- Master (red): a commit hinted with one company makes B load `*`: 1
load, 801 rows.
- This branch (green): B loads only `company-7`: 1 load, 41 rows. A
commit with two hints loads only those two companies (80 rows). An
unscoped commit still loads all 801 rows.
- `cd packages/db && npx vitest run src/work-signals.test.ts` (32 tests,
real PostgreSQL). New tests:
- deduplicated hints per topic, including a released savepoint and a
rolled-back savepoint
- split into payloads of at most 1,024 bytes, at most 7,000 bytes in
total, and the union of hints is complete
- overflow and one oversized hint fall back to one unscoped payload
- malformed hint lists and unknown versions wake unscoped; unusable
payloads do not wake; own-origin payloads are dropped; nothing throws
- an old-format payload `{version, origin, topic}` gives an unscoped
wake
- every new payload passes a copy of the receiver from master, which
reads it as an unscoped wake for the correct topic
- the debug event logs the received hints
- With master's `work-signals.ts`, 5 of these tests fail (red). With
this branch, all pass.
- Also run: `scoped-deadline-work.test.ts`,
`delivery-work-coordinator.test.ts`, `deployment-notifications.test.ts`,
`run-start-handover.test.ts`, `heartbeat-run-start-handover.test.ts`,
`heartbeat-run-event-atomic-append.test.ts` (111 tests pass),
`work-signal-debug.test.ts` (12 pass), and `tsc --noEmit` for
`packages/db`.
## Risks
- Mixed deploys, new sender to old receiver: every payload is version 1
and at most 1,024 bytes. The old receiver ignores the `hints` field and
wakes unscoped, as today. A commit with many hints sends several
payloads, so an old receiver can get up to about 7 unscoped wakes for
one commit. The coordinator merges them into one `*` reload.
- The sender bounds bytes, and old receivers compare `.length` in UTF-16
code units. A string never has more UTF-16 code units than UTF-8 bytes,
so a payload of at most 1,024 bytes always passes the old check.
- PostgreSQL collapses identical NOTIFY payloads in one transaction. The
chunks of one topic always carry different hints, so PostgreSQL drops
none of them.
- Mixed deploys, old sender to new receiver: the old payload has no
`hints`, so the new receiver wakes unscoped, as today.
- I did not use one large payload of about 7 KB: current receivers drop
any payload over 1,024 characters before they parse it, so they would
miss the wake during a deploy overlap.
- Consumers that read hints (for example `chat-action-work`,
`approval-queue-seeding-work`) now get the same hints from another
process that they already get from a local commit. Before, they got no
hints from another process.
- The receiver accepts hint strings from the database channel. Consumers
decode them in the same way as local hints. `scoped-deadline-work`
already turns a decode failure into a `*` retry.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edit, test execution).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
819f61f1fd |
fix(evals): repair four cohort-15 chat cases and prefix board links in completion feedback (#21682)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The chat-feature eval catalog runs real chat journeys. It grades saved state, rendered links and run ledgers. > - Chat-eval cohort 15 (OpenCode with `openrouter/deepseek/deepseek-v4.1-flash-1017`, master `36101ebcb9`) failed four cases where the agent did the right thing or where our own server text misled it. > - One failure was a harness race introduced by #21667. Two failures came from graders that read free-form text with keyword regexes. One failure came from an unprefixed approval link in the server's completion feedback, plus a grader that required a literal prefix. > - This pull request fixes the harness step, replaces the free-form graders with saved-state and clicked-link checks, and makes the server emit company-prefixed board links. > - The benefit is that these cases measure the feature again. The retained correct outcomes pass, and the relevant broken outcomes still fail. ## Linked Issues or Issue Description Refs #21667 **What happened?** In chat-eval cohort 15, four cases failed: - `projects.share-project`: the harness followed the agent's ID-based project link. The app then rewrote the URL to the project slug and redrew the page. The owner step timed out because Riley's option in the people picker was "detached from the DOM". #21667 introduced this path. - `skills.author-skill`: the saved skill was a correct, richer checklist. Per-sentence keyword regexes rejected it. For example, "ignore the mouse" tripped the forbidden word "ignore". - `inbox-search.show-decisions`: the reply linked the right approval and tasks and said the question was "addressed to a different teammate". A regex required the word "Riley". - `agents.hire-needs-approval`: both replies linked `/approvals/<id>` without the company prefix. The agent copied that link from `server/src/services/native-runtime/native-completion-feedback.ts`. The grader required a literal `/<prefix>/approvals/<id>`, although the board redirects the unprefixed route. **Expected behavior** Correct agent behaviour passes. Grading uses saved state, setup literals and rendered or clicked links, not free-form wording. The server gives agents canonical company-prefixed board links. **Steps to reproduce** 1. Use the retained cohort-15 evidence for the four cases. 2. Run the old graders on the retained skill markdown and decisions reply. Both fail. 3. For share-project, open a project configuration page by ID while the app rewrites the route after a delay, then use the access picker before the rewrite completes. 4. For hire-needs-approval, finish a task with a pending linked approval. The feedback links `/approvals/<id>`. **Paperclip version or commit** master `36101ebcb9` (cohort 15); fixed on top of current master. ## What Changed - `projects.share-project` (`internal-privacy.ts`, `runtime-support.ts`): after the harness follows the agent's link, it waits until the URL settles on the project's canonical route key. This reuses the #21667 approach for agent profiles: `settleAgentProfileRoute` now calls a shared `settleRouteKey`, and `projectRouteKey` follows the app's `projectRouteRef`. The owner's picker selection (`addProjectAccessMember`) now retries as one unit. If the dialog or picker closes, or an option detaches, it reopens them, searches again and selects again. It continues only when the picker shows the person and Add is enabled. - `skills.author-skill` (`integrator-skills.ts`): removed the keyword-regex checklist grader, the waiver regexes, and the reply regexes for "global activation" and "attached to me" claims. The saved-state checks stay: one agent-created writable skill, its current revision authored by the responding agent, the same bytes on the skill and the revision, the Skills/Studio inspection with reload, and assignment scope from saved assignments and audit rows. The checklist check now requires the setup literals `375`, `WCAG` and `AA` as whole tokens in the saved SKILL.md and in the rendered editor. Whether the skill still asks for keyboard-only navigation and visible labels, and whether it waives a check, is left to human review. Oracle version `skill-authoring.v10`. - `inbox-search.show-decisions` (`internal-decision-search.ts`): removed all reply regexes, including the `Riley` requirement and the wording-based separation checks. The grade now uses saved state (pending decisions unchanged, answered history unchanged, each person's own attention queue and resolver) and rendered links to the exact approval, the question's task and the history task. Oracle version `decision-search.v3`. - Product (`native-completion-feedback.ts`): the completion feedback now links `/<prefix>/approvals/<id>` and `/<prefix>/issues/<identifier>`. This fixes the approval link and the four sibling task links in the same file. - `agents.hire-needs-approval` (`runtime-hiring.ts`): the harness now clicks each reply's rendered link to its approval and records where the UI lands. The grade accepts a prefixed link, or an unprefixed link that the board redirect lands on `/<prefix>/approvals/<id>` with the proposed hire visible. Without such a link, the harness still opens the approval directly so later phases keep their evidence, and the grade fails. Oracle version `runtime-hiring.v4`. - No eval YAML changed, so the TestOps manifest does not change. `chat-catalog-descriptors.json` records the three oracle version bumps. ## Verification - Red/green tests (each new test fails on the old code and passes on the new code): - `internal-privacy-share-route.test.ts` (real Chromium): the project page rewrites its route after 0, 400 and 1500 ms, and one case closes the picker while its first results render. With the old flow, the 400 ms rewrite and the closed picker both time out on the option click (the cohort-15 failure). With the fix, Riley is added on the settled route in all cases. - `integrator-skills.test.ts`: the retained cohort-15 SKILL.md passes. The old grader rejects it. A stale skill body, another author, and a lost 375 width still fail. - `internal-decision-search.test.ts`: the retained cohort-15 reply and a reworded reply pass. The old grader rejects the retained reply. A missing or wrong approval, question-task or history-task link fails. - `native-completion-feedback.test.ts`: a pending linked approval produces `[the pending approval](/<prefix>/approvals/<id>)`, and a pending review stage produces a prefixed task link. The old code produces `/approvals/<id>`. - `runtime-hiring.test.ts`: unprefixed links that the redirect lands on the right approval pass. The old grader rejects them. A link that did not land, landed on another approval, was not clicked, or is another href still fails. - `npx vitest run --config tests/runner-e2e/vitest.config.ts chat-catalog`: all passed (one negative control in `internal.test.ts` changed from a reply wording to "approved while listing"). - `pnpm test:e2e:runner:typecheck` and `server` `tsc --noEmit`: passed. - `server` vitest for `native-completion-feedback`, `native-current-wake-completion-feedback` and `heartbeat/run-completion`: 201 tests passed. - Narrow paid run of the four full case ids on this branch: run 43657390762 (results are added in a comment). ## Risks - Low risk. The product change only adds the company prefix to links in agent feedback text. It adds one indexed company read, and only on the branches that return a link. - Less is graded automatically, by design. `skills.author-skill` no longer checks that the skill keeps the keyboard-only and visible-label checks or that it does not waive a check. `inbox-search.show-decisions` no longer checks how the reply names resolvers or separates the person's decision from others' and from history. `skills.author-skill` no longer checks whether the reply describes the assignment scope truthfully. A person reviews this meaning. Saved state and links still catch the plausible broken outcomes listed above. - `agents.hire-needs-approval` now needs a reply link that really opens the approval. A reply without a link to its approval fails as before. ## Model Used - Claude, model ID `claude-opus-5-5` (Claude Code agent with tool use and code execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
ca23330c69 |
fix(recovery): continue a task on its own when its saved files are permanently lost (#21541)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a run stops early, Paperclip recovers its saved files and then continues the task automatically > - Sometimes the sandbox is gone. Then saved-work recovery ends with `source_unavailable` (use another strategy) > - In that state the task showed Retry. Retry worked, because it continues in a fresh workspace. But nothing continued the task automatically > - A task must never need a person's Retry for a system failure. The target is 0% > - This pull request lets the system take the same fresh-workspace step that Retry takes, when the saved files are permanently lost > - The benefit is that these tasks continue on their own, and the sweep reports them correctly ## Linked Issues or Issue Description Refs #21478 **What happened?** A legacy run on a remote sandbox failed in setup. Its unsaved files were held for saved-work recovery. The sandbox was gone, so the recovery ended `source_unavailable` / `use_alternative_strategy`. The task then showed Retry. For about 2 hours, the assigned-work sweep logged `retry_offered` with outcome `not_started` every 15 minutes. A person had to press Retry. **Root cause** The fresh-continuation fallback returned `person_owned` / `retry_needed`. It does this when a later run of the task was withheld at dispatch (here, a scheduled retry that the hold blocked) and the source run does not continue automatically. The source was a person's earlier Retry that failed with `setup_failed`, so it was not an automatic recovery run and not a restart-class failure. Without the withheld run, the same task already continued automatically. Restore-hold ownership (`isOwnedReconciliationHold`) and the legacy refusal in `inspectFallback` were not the blocker. **Expected behavior** When saved-work recovery proves that the files are permanently gone, the system continues the task in a fresh workspace on its own. This is the same path that Retry takes. ## What Changed - `fresh-continuation-fallback.ts`: the eligibility rule (`interruptionEligibility`, which the fallback and the backlog pass share) also accepts a system-owned source run when a hold of that run has saved-work recovery `exhausted` with a permanent unverifiable-source failure (`UNVERIFIABLE_SOURCE_CODES` × `PERMANENT_SOURCE_RETRY_CLASSES`). - `recovery/assigned-work-sweep.ts`: admission can settle the hold and coalesce the sweep's wake into a pending fresh continuation. The sweep now reports this as `repaired` / `queued` (`continuedFromSavedWork`). Before, it reported `enqueue_refused` / `not_started`. - `legacy-execution-recovery.ts`: the stranded sweep calls `terminalizeLegacyExecution` for the same failed run on every pass. Its restore-hold branch rewrote the hold every time, with a new `automaticRecovery.recordedAt`, a new `updatedAt`, and `continuationDelivery: "invalidated"`. The branch now returns early when the hold already records the same no-replay decision for this run, or when a fresh continuation took the hold over. Finding: a continuation that was already recorded was not clobbered, because the existing `executionReconciliation.runId` guard returned early. But a hold with a `coalesced` delivery state would have been reset, and the churn kept bumping `updatedAt`. - New regression test `legacy-source-unavailable-continuation.test.ts`. The existing `assigned-work-invariant` test now expects the corrected sweep outcome. ## Verification - Red/green: `legacy-source-unavailable-continuation.test.ts` uses `failedRemotePreparation()` with a `destroyed` termination receipt, `recoverRetainedWorkspace`, a withheld scheduled retry, and a source that was a person's Retry. Then it runs `assignedWorkSweep(...).checkIssue` with the real `heartbeatService(db).wakeup`. - Without the fix: outcome `enqueue_refused`, and the fallback returns `person_owned` / `retry_needed`. - With the fix: outcome `repaired`. A stranded pass keeps `continuationDelivery: "pending"`. `deliverReconciledExecutions` starts the successor run. The task no longer shows Retry. There is no `not_started` entry. - Red/green: the "revisits" test fails without the `terminalizeLegacyExecution` guard, because the hold evidence changes. It passes with the guard. - Locally passing: `fresh-continuation-fallback`, `fresh-continuation-move-forward-offer`, `fresh-continuation-retry-delivery`, `legacy-execution-recovery`, both `assigned-work-invariant` suites, `assigned-work-stalled-queued-run`, `assigned-work-sweep-round-trips`, `saved-work-recovery-overdue`, and the new test file (10 files, 225 tests). `tsc --noEmit` passes for `server`. - Not run locally: the full suite. CI covers it. ## Risks - Behavior shift: a task whose source failure is system-owned now continues automatically after its saved files are proven permanently gone, even when an earlier wake was withheld. The new run does not replay the old run. It starts in a fresh workspace, with the same notes and quarantine that Retry uses. Person-owned failure codes still wait for a person. - The new hold guard can skip a rewrite. It skips only when the hold is already resolved with the same no-replay decision and the same restore-recovery data. A changed decision is still written; the test covers this. - No schema changes. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, local Vitest against embedded Postgres). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>canary/v2026.1115.0-canary.10 |
||
|
|
68f641ba0b |
fix(recovery): continue a native run cut off by a sandbox client timeout (#21736)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The native runtime runs an agent's provider turn in a sandbox (for example Daytona) and records each failure on the run and its native coordinator > - A Daytona SDK request can hit its client-side deadline during a provider turn. The run then fails with `native_session_interrupted` in an ambiguous state, because the turn started and its outcome is unknown > - That failure blocks the task under a board-owned watchdog hold. No recovery owner selected the hold, because its cause is not an execution-reconciliation cause. The task never ran again, and only a person's Retry could continue it > - Recovery from an infrastructure fault must work automatically, without a person > - This pull request lets the execution-disposition pass hand that hold to the existing fresh-continuation fallback > - The benefit is that the task continues once, in an isolated fresh run, and the cut-off turn is never replayed ## Linked Issues or Issue Description **What happened?** A native run on a Daytona execution environment (native runtime, an OpenRouter model) started its provider turn. A Daytona request then failed with `HTTP request timed out after 300000ms waiting for a response. This is a client-side deadline (DaytonaConfig.requestTimeoutMs, or the per-call operation/execution timeout); any operation already started on the server may still be running.` The run failed with `native_session_interrupted`. The task had no further run: no automatic retry and no continuation. **Expected behavior** The task continues automatically, exactly once, in a fresh run from its saved work. The cut-off provider turn is not replayed. **Steps to reproduce** 1. Start a native run for an assigned task. Let its provider turn start (a `turn.started` event is saved). 2. Make a sandbox request in that turn fail with the Daytona client-side timeout error. 3. The executor saves the coordinator as `terminal_failure` with `recoveryMode: ambiguous_state`, blocks the task, and writes an active board hold with cause `native_session_interrupted`. 4. Run the execution-disposition worker (`settleUnrecoverableExecutions`). Before this change, nothing happens and the task stays blocked. **Paperclip version or commit** `master` atcanary/v2026.1115.0-canary.9 |
||
|
|
01c1e708e9 |
fix(runner): stop a turn that keeps resending a refused tool call and continue fresh (#21734)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native agent runs call Paperclip tools, and
`RepeatedToolFailureGuard` refuses an exact repeat of a call that
already failed validation twice
> - The refusals are not counted and never end the turn, so a model
whose host drops an argument can resend the same refused call until the
run's wall clock
> - In a recorded restart-chaos wave, an opencode/DeepSeek agent sent
the same `PATCH /api/issues/{id}` through `call_api` 431 times in 40
minutes without `body`; 427 were refused. The run held an agent slot and
delayed other tasks
> - The system knows the next step (stop the turn and continue on a
fresh session), so the harness must own it, with no Retry banner
> - This pull request counts refusals, stops the provider turn after a
bounded number with the classified cause `agent_tool_call_loop`, and
queues an automatic fresh-session continuation that tells the model to
use a dedicated tool
> - It also records the declared argument names that each opencode tool
call delivered, so we can tell whether the bridge or the model drops
`body`
> - The benefit is that a looping agent frees its slot in seconds
instead of 40 minutes, and the task continues by itself
## Linked Issues or Issue Description
**What happened?**
An opencode agent on DeepSeek created two subtasks and then tried to set
their descriptions with `call_api` `PATCH /api/issues/{id}`. Every call
arrived with `operationId`, `pathParams`, `query` and `responseText`,
but without `body`. The guard refused each repeat. The model said "I
need to include body" more than 20 times and kept sending the same call:
431 calls, 427 refused, over 40 minutes. The run stayed active and its
agent slot delayed other queued work.
**Expected behavior**
After a small, bounded number of refusals, Paperclip stops the provider
turn and continues the task automatically on a fresh session, with a
hint to use a dedicated Paperclip tool. The board shows no Retry banner
because the system owns this recovery.
**Steps to reproduce**
1. Run a native agent whose model resends one tool call with a missing
required argument.
2. The guard refuses the third and later identical calls.
3. On master, the refusals never end the turn.
**Paperclip version or commit**
`master` at
canary/v2026.1115.0-canary.8
|
||
|
|
c7efca1eaf |
perf(recovery): read an agent's own reporting chain and each pass's review waits once (#21724)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The server runs its background recovery queues on one Node main thread. Each database statement and each byte returned costs main-thread time. The target is 1,000 concurrent agents per server. > - On the idle hosted staging instance, the stranded-task check is the top remaining cost. Each check read the full agent row (with its configuration blobs) and then every agent of the company, only to decide whether this one agent can run. Per check it also opened a repeatable-read budget transaction for the same company and agent. > - The org-chain evaluation only looks at the agent's own managers. Reading the whole company is work proportional to the size of the company, for an answer that depends on a few rows. > - The same check also read the external review wait of each in-review task one task at a time. > - This pull request reads only the agent's reporting chain (one recursive query up `reports_to`), gives the stranded check a lean agent read, and reads the review waits of a pass's tasks in one call. > - The benefit is half the bytes per check (17.2 KB to 8.6 KB), 87 percent fewer rows, and no per-task review-wait chain for tasks that wait, with the same answers. ## Linked Issues or Issue Description No issue exists. This describes the problem with the fields of `.github/ISSUE_TEMPLATE/enhancement.yml`. **What existing behavior does this improve?** The per-task reads of the stranded-task recovery check (`reconcileStrandedAssignedIssues`) and of every caller of `evaluateAgentInvokabilityFromDb` and `agentService.getById`. **Subsystem affected** Server: issue recovery and agent invokability. **Current behavior** `evaluateAgentInvokabilityFromDb` reads `id, company_id, name, reports_to, status` for every agent of the company on each call. `getById` does the same for `orgChainHealth`. The stranded check calls it once per task, after a full-row agent read with the configuration blobs. On staging at idle these are the `agents where company_id = $1` read at 4.7/s and the full `agents where id = $1` read at 4.2/s. The check also calls `listExternalReviewWaits` for one task at a time (the `chat_message_links` / `chat_github_reviews` reads at about 3/s on staging). **Proposed behavior** Read only the agent's managers, with the same cycle, missing-manager, cross-company and terminated rules. Read nothing when the agent has no manager or when its own status already decides the answer. Read the stranded check's agent with only the fields it uses. Read the review waits of all tasks of a pass in one call, and use that answer only to skip a task that waits. **Reason and benefit** At idle, the average result size on staging is about 17 KB per statement. The stranded check is about 22 statements and 17 KB per task, and it runs about 3 times per second there. ## What Changed - `server/src/lib/agent-reporting-chain.ts` (new): `readAgentReportingChain` reads the agent's managers with one recursive CTE up `reports_to`, limited to the agent's company. `UNION` stops the walk at a cycle. A root agent costs no statement. - `server/src/services/agent-invokability.ts`: `evaluateAgentInvokabilityFromDb` evaluates against the chain, not the company list. It reads nothing when the agent's own status is not invokable, because the status then decides the answer and the chain is not part of it. Every caller benefits with no change: wake enqueue, queued-run claim, retries, run dispatch, wake-queue promotion, continuation delivery, task watchdogs, thread interactions, native completion feedback, native session resumption, native review participants, agent conversations and recovery. - `server/src/lib/agent-records.ts`: `getById` computes `orgChainHealth` from the chain, not from the company list. - `server/src/services/recovery/service.ts`, stranded check: - The agent read selects only `id, company_id, name, reports_to, status, lifecycle_state, lifecycle_holds`, not the full row. - `hasPersistedDurableWaitPath` takes the pass. When the task is in the pass, the first in-review task reads the external review waits of every task of the pass with one `listExternalReviewWaits` call. A task in that answer waits and is skipped. A task not in that answer is read again alone, exactly as before, so a missing answer never lets a check go further than a fresh read would. - A company-wide call without a pass gets a pass of its in-review candidates. - `server/src/services/scoped-deadline-work.ts`: an optional `beforeChecks(entries)` hook gets the entries of the batch before their checks run. - `server/src/services/recovery/pass-reads.ts` (new) and `server/src/services/recovery/work.ts`: the task queues build a pass from the batch in `beforeChecks` and give it to each check of that batch. - Tests: - `server/src/__tests__/agent-reporting-chain.test.ts` (new): property test, see Verification. - `server/src/__tests__/issue-recovery-pass-reads.test.ts` (new): statements and bytes per check, per pass and per full worker drain. - `server/src/__tests__/issue-recovery-review-wait-pass.test.ts` (new): review-wait answers and reads, see Verification. - `server/src/__tests__/monthly-spend-service.test.ts`: the select-sequence stub no longer serves a company agent list to `getById`. - `server/src/__tests__/helpers/recovery-population.ts` (new): seeds a company with an org chart, review tasks with succeeded native runs, and optional paused, terminated and recovery-action tasks. - `server/src/__tests__/issue-recovery.test-fixture.ts`: an optional statement meter, installed before the first connection. - `server/src/__tests__/agent-record-read-projections.test.ts`: `getById` now makes one typed `agents` select; the chain is a raw query. ### Why the shared review-wait read cannot start or escalate work that a fresh read would not - A pass is one coordinator batch: at most 25 checks, one after the other. Its read is never older than the start of the batch. Nothing is kept across batches. - Only the positive answer is used: "this task waits on an external review or reply", which leads to a skip. A skip only delays the task to its next check, and the end of a wait sends its own hint. - A negative answer is not used. The task is read again alone, so every decision that goes further (an enqueue or an escalation) uses a fresh read, as today. - A first version also kept the agent's invokability and the budget answer for the pass. Review found that a kept permissive answer can reach an escalation, which does not re-check those facts the way an enqueue does. That reuse is removed: those reads stay per task. No authorization, pause, approval or budget check is removed or reordered. ## Verification - Equality proof (`agent-reporting-chain.test.ts`, embedded PostgreSQL): - 60 generated org charts, 2 companies each, up to 45 agents. Shapes: deep chains (depth 44), trees, full cycles and mixed graphs. - Managers are missing (inserted with foreign keys off), in another company, terminated, paused or pending approval at random depths. Agent statuses include an unknown value. - For each agent, the test compares the old list-based result with the new result. `evaluateAgentInvokabilityFromDb` deep-equals `evaluateAgentInvokability(agent, companyRows)`. `getAgentWorkEligibility` on the chain deep-equals it on the company list (this is the `getById` path). The test also asserts that every block reason and every chain-health reason occurred. - The list-based evaluation has no depth cap (`reporting_chain_too_deep` is declared but never returned), so the chain read has none either. A cycle ends the walk in both. - Mutation check: when the recursive step does not walk up, both tests fail. - Review-wait equality (`issue-recovery-review-wait-pass.test.ts`, embedded PostgreSQL): two tasks with a direct chat reply wait, one task with an inherited GitHub review wait, one task with none, and one task with an inherited GitHub review wait in another company. One call for all of them gives the same per-task answer as one call per task, for both companies. - Review-wait reads per pass (same file): the pass makes 2 bound-run message reads (one shared, one for the task whose candidate is not a wait) where per-task reads make 3, and 89 statements instead of 91. The check results are identical. On master the pass is ignored, so the test fails (3 is not less than 3). - Statements and bytes (`issue-recovery-pass-reads.test.ts`, 50 agents, 200 review tasks with succeeded native runs and no review waits, embedded PostgreSQL, statement meter on the client sockets): | | master | this branch | | --- | --- | --- | | One check: statements | 22 | 22 | | One check: bytes received | 17,170 | 8,617 | | One check: rows | 57 | 8 | | 25 checks: bytes | 429,331 | 215,505 | | Real worker, full company drain (200 checks): statements | 4,414 | 4,422 | | Real worker, full company drain: bytes | 3,455,957 | 1,749,701 | | Real worker, full company drain: rows | 11,600 | 1,800 | | Company agent-list reads per drain | 200 | 0 | Here no task waits, so each batch pays one extra review-wait statement (8 per drain). Where tasks wait, the shared read replaces the per-task review, publication and authorization reads. - Regression after the review change, embedded PostgreSQL (664 tests): scoped-deadline-work, heartbeat-process-recovery, issue-recovery-deadlines, assigned-work-invariant, recovery-deadline-acceptance, heartbeat-issue-liveness-escalation, slack-conversation-lifecycle. Also issue-list-review-wait-authorization, external-review-wait, monthly-spend-service. - Regression of the first version, embedded PostgreSQL (757 tests): issue-assignee-invokability-routes, agent-record-read-projections, issue-graph-liveness-equivalence, heartbeat-paused-assignee-new-task, heartbeat-process-recovery, heartbeat-issue-liveness-escalation, assigned-work-invariant, recovery-deadline-acceptance, issue-recovery-actions, heartbeat-refused-assignment-wake, failed-continuation-continues-again, agent-invokability, agent-cross-tenant-authz-routes, heartbeat-stop-queued-task, issue-tree-pause-resume-run. - `tsc --noEmit -p server/tsconfig.json` passes. ## Risks - A test double that gives `evaluateAgentInvokabilityFromDb` an object with only `select` now needs `execute`, and a select-sequence stub of `getById` gets one select fewer. CI found one (`monthly-spend-service`), fixed here. - A batch with no waiting task pays one extra cheap review-wait statement. - This branch and #21729 both touch `server/src/services/recovery/work.ts` in nearby lines. The second one to merge may need a small rebase. - `server/src/services/recovery/service.ts` was already past 1,000 lines. This change adds about 25 lines to it, beside the check it changes. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, local PostgreSQL test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
7c0c116edf |
test(e2e): accept every dependency wake producer for the one automatic resume (#21732)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a blocker task is completed, Paperclip wakes the blocked dependent task automatically > - Two producers emit that `issue_blockers_resolved` wake: the issue update route and the issue-graph liveness backstop (plus workspace finalize); they share one idempotency key and coalesce into one wake run > - The QA task-dependency-resume journey requires the run's `contextSnapshot.source` to be exactly `issue.blockers_resolved` > - On QA both producers emitted 43 ms apart; the run was created once, but the later backstop wake recorded its source on the run, so the journey failed a resume that happened exactly once > - This pull request accepts any of the three producers and keeps every other resume condition > - The benefit is that the journey tests the user-visible promise (one automatic resume caused by this blocker) instead of which internal producer won a race ## Linked Issues or Issue Description **What happened?** QA run (launch 3937) failed task-dependency-resume on `dependency_one_automatic_resume`. The dependent's activity log shows `issue.blockers_resolved_wake_emitted` from `issue_update` at 06:59:39.973 and from `issue_graph_liveness_backstop` at 06:59:40.016 with the same idempotency key and the same wake run. Exactly one run executed, started after the blocker completed, and succeeded. **Expected behavior** The journey accepts one automatic, blocker-caused resume regardless of which producer's label the coalesced run kept. **Steps to reproduce** Grade the retained evidence with `contextSnapshot.source = "issue_graph_liveness.backstop"`: `dependency-one-automatic-resume` fails although every other condition holds. ## What Changed - `task-dependency-resume-evidence.ts`: the wake source must be one of `issue.blockers_resolved`, `issue_graph_liveness.backstop`, `workspace.finalize` (the producers of this wake reason in `server/src`). - Still required: one dependent run, `invocationSource` automation / `triggerDetail` system, wake reason `issue_blockers_resolved`, the resolved blocker id, start after blocker completion, committed native run, no leftover checkout. ## Verification - Red/green: new cases "accepts the same automatic resume when the issue_graph_liveness.backstop / workspace.finalize dependency wake recorded its source" fail before and pass after. - Existing mutants still fail: "mutation name used as wake source" (`issue.blocker_done`) and "missing saved wake source". - `tests/runner-e2e` task-dependency tests: 3 files, 108 tests pass. ## Risks - Low risk: journey grading only. Separate note: the coalesced run keeps the last producer's source label; whether it should keep the first is a provenance question, not addressed here. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
41a3520a2f |
fix(issues): count a blocker covered when its wake already created a queued run (#21731)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A blocked parent task shows "covered" when its blockers have work
that will start by itself, and "needs attention" when a person must act
> - The blocker-attention read counts a blocker as active only through
its current execution run, or through a waiting wake that has not
created a run yet
> - A child whose wake already created a queued run that has not started
matches neither, so the parent shows "needs attention" and lands in the
blocked inbox
> - That work starts by itself when an agent slot frees, so the
attention is false and asks a person to act for no reason
> - This pull request also counts a blocker as active when its waiting
wake points at a queued or running run
> - The benefit is that the board and the blocked inbox stop showing
false attention while queued child work waits for a slot
## Linked Issues or Issue Description
**What happened?**
A parent task was blocked by two child tasks. Each child had a queued
run that a wake had created, and the runs waited for agent slots. The
parent showed `needs_attention` / `attention_required` and appeared in
the blocked inbox. The children started about 25 minutes later and the
parent then finished without a person.
**Expected behavior**
The parent shows `covered` with reason `active_child` while a child's
queued run waits to start.
**Steps to reproduce**
1. Create a blocked parent task with one child blocker that is assigned
to an agent.
2. Insert a queued wake for the child whose `runId` points at a queued
`heartbeat_runs` row, and leave the child's `executionRunId` empty.
3. List blocked tasks. The parent shows `needs_attention`.
**Paperclip version or commit**
`master` at
|
||
|
|
9ebc276b42 |
test(restart-chaos): let R5 wait out a slow model request and stop calling held wakes undispatched (#21730)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart-chaos harness (`tests/restart-chaos`) grades each task of a chaos wave with rules R1–R6 > - R5 fails a run that stays `running` with no new evidence for more than 3 minutes (`silent_dead_run`) > - In wave w-20261115T044649-s7106, task t028 failed R5. Its run was quiet for 303.6 s (05:22:13.005 to 05:27:16.647) because one DeepSeek request through OpenRouter was slow. Every sample showed an active provider turn and no tool running. The run then finished normally. > - A slow model request is not a dead run. False failures hide real failures and make the wave number wrong. > - This pull request lets R5 wait up to 10 minutes while a provider turn is active and no tool runs. It also stops the release-swap grade from calling queued wakes "never dispatched" while a live run holds them. > - The benefit is a wave grade that counts only real dead runs and real lost wakes. ## Linked Issues or Issue Description **Bug description** R5 reports `silent_dead_run` for a run that waits on one slow model request. The samples show `providerTurnActive=true` and no tool in flight for the whole gap. The run then finishes normally. Separately, the release-swap grade detail says "2 queued wake(s) never dispatched" for a task whose run was still running. Those wakes wait behind the active run by design. **Steps to reproduce** Grade the t028 samples of wave w-20261115T044649-s7106 with `gradeR5`. The result is `fail: silent_dead_run 7b51b352… 182s`. Grade a release-swap task that has a stale running run and queued wakes. The detail contains "never dispatched". **Expected behavior** R5 passes a quiet run while a provider turn is active and no tool runs, up to 10 minutes. R5 still fails a quiet run with no active provider turn after 3 minutes, and a provider turn quiet for more than 10 minutes. The grade does not call wakes undispatched while a queued or running run holds them. ## What Changed - `grader.ts`: add `PROVIDER_TURN_LIMIT_MS` (10 minutes). `deadRunningRun` uses this limit instead of the 3-minute `STALL_LIMIT_MS` when the run sample has `providerTurnActive`. A run with a tool in flight or a live process is still skipped as before. - Why 10 minutes: the observed slow request took 303.6 s. Common provider HTTP clients (the OpenAI and Anthropic SDKs) time out a request at 10 minutes. After 10 minutes the provider client has given up, so continued quiet means the run is dead, not slow. A turn whose process died also keeps `providerTurnActive` set, so the bound still catches it. - `release-swap-grade.ts`: omit the "N queued wake(s) never dispatched" note while the task has a queued, running or scheduled-retry run. The stale-run note stays. - New fixture `fixtures/t028-slow-provider-turn.json`: the real t028 samples from 05:21:30 to the run's end (116 samples). - `README.md`: document the provider-turn bound for R5 and the queued-wake rule. ## Verification - Red on `origin/master`, green with this change: - `triggers.test.ts` › R5 live evidence › "does not flag a slow model request while the provider turn is active with no tool running (w-20261115T044649-s7106/t028)" - `triggers.test.ts` › R5 live evidence › "still fails a silent run with no active provider turn, and a provider turn silent past the provider-latency bound" (guards: 4 min quiet with no provider turn fails; 9.5 min quiet with a provider turn passes; 10.5 min quiet with a provider turn fails) - `release-swap-grade.test.ts` › "does not report queued wakes as never dispatched while a run holds them (w-20261115T044649-s7106/t019)" (also checks that wakes with no live run still get the note) - `npx vitest run --config tests/restart-chaos/vitest.config.ts`: 71 files, 826 tests pass. - `npx tsc -p tests/restart-chaos/tsconfig.json --noEmit`: clean. - No wave was run. ## Risks - Low risk. The change touches only the test harness grader. - A run whose provider turn is active but whose process is truly dead now fails R5 after 10 minutes instead of 3. The fault is still caught, but later. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used in Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>canary/v2026.1115.0-canary.7 nightly/v2026.1115.0-nightly.0 |
||
|
|
6400698a8e |
test(e2e): find the native Plan chip in the Coordinator chat thread (#21727)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The QA Cloud plan-revise-accept journey checks that a person can
revise and approve an agent's Plan
> - It first checks that the saved Plan chip is visible, scoped to a
settled `task-chat-turn` so a live "Writing plan" chip cannot count
> - The Coordinator workspace renders the Plan chip directly in the chat
thread; there is no `task-chat-turn` element and no `data-settled`
attribute
> - So the journey never finds the chip and fails before the person
reviews the plan
> - This pull request also accepts the chip in the chat thread
> - The benefit is that the plan journey reaches the revision and
approval steps it exists to test
## Linked Issues or Issue Description
**What happened?**
On QA Cloud (build
|
||
|
|
61c315ebab |
perf(chat): keep GitHub review check deadlines on company-wide rebuilds (#21725)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The GitHub chat channel shows a check run on each pull request it reviews. The review-check sweep (`review-check-work.ts`) keeps one `github_review_check` action per check. It republishes the check when its status changes (queued, running, completed) and polls the pull request head on a saved deadline. > - Each time the sweep loads its checks, it computes the current check status of every `received` check (`githubReviewCheckState`: 3–5 statements) to see whether the status changed. > - Run lifecycle changes anywhere in a company send a company-wide hint. So the sweep rebuilds every company's checks every 30 s and recomputes the status of every waiting check, even when nothing about those checks changed. > - This pull request keeps the saved poll deadline on a company-wide rebuild and reads the status only on targeted hints, at startup, and in `check()`. > - The benefit is that an idle company with many waiting checks costs one page read per rebuild instead of 3–5 statements per check. ## Linked Issues or Issue Description Refs #21705, #21711 (the same class of idle reload in the GitHub response sweeps). No issue exists. This describes the problem with the fields of `.github/ISSUE_TEMPLATE/bug_report.yml`. **What happened?** On paperclip.staging atcanary/v2026.1115.0-canary.6 |
||
|
|
5acf27bc3b |
perf(server): read the instance role and membership once per GET request in authorization (#21716)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Every board request goes through authorization, and the server runs
all requests on one Node main thread
> - The cost on that thread is per database statement
> - One `GET /api/issues/:id` makes several authorization decisions (the
issue, its project, mentioned projects, related task edges), and each
decision reads `instance_user_roles` and `company_memberships` again
> - On a task with a blocker and a project mention, that is 4 role reads
and 4 membership reads in the authorization service for one request, all
with the same answer
> - This pull request shares those two reads among the decisions of one
GET or HEAD request
> - The benefit is fewer statements on every polled board read, with no
authorization cache across requests
## Linked Issues or Issue Description
**What existing behavior does this improve?**
`authorizationService.isInstanceAdmin` and `getActiveMembership` read
the database for each decision in a request.
**Subsystem affected**
Server: authorization service and app middleware.
**Current behavior**
`GET /api/issues/:id` for a plain task sends 3 `instance_user_roles`
reads and 3 `company_memberships` reads (1 each in the session
middleware, 2 each in authorization). A richer task sends 5 and 5.
**Proposed behavior**
A new middleware, `authorizationReadScope`, opens a per-request memo for
GET and HEAD requests. For a session actor, it seeds the instance admin
answer that the session middleware read at the start of the same
request. Inside the scope, `isInstanceAdmin`, `getActiveMembership`,
`issueReadSqlCondition` and `projectReadSqlCondition` read each key
once. The plain task now sends 1 role read (the session middleware) and
2 membership reads (1 in the session middleware, 1 in authorization).
The issue list and summaries routes also drop their extra role reads.
Other methods, background work and other callers keep the old reads. The
memo ends with the request, so a revoked membership takes effect on the
very next request.
**Reason and benefit**
Fewer statements per polled read, with the same decisions.
**Breaking changes**
None.
## Memo lifetime rule
- The memo exists only for GET and HEAD requests.
- It wraps `res.writeHead`, `res.flushHeaders` and `res.write` once and
never reassigns them, so hooks that later middleware installs on the
response stay in place.
- It closes at the first of: `res.writeHead`, `res.flushHeaders`,
`res.write`, `finish` or `close`. After it closes, every read goes to
the database. A stream, a long poll, a download or a callback that runs
after the response therefore never uses the memo.
- Each entry is used for at most 2 seconds
(`AUTHORIZATION_READ_MAX_AGE_MS`), then read again. Correctness does not
depend on a list of routes.
- A rejected read is removed from the memo, so the next decision reads
again.
- Nothing is kept across requests.
Long-lived and streaming routes that I checked:
- `GET /plugins/:pluginId/bridge/stream/:channel` (SSE): authorizes once
at connect with `assertBoardOrgAccess` and `assertCompanyAccess` on
`req.actor`. It does not re-authorize per event. The memo closes at
`writeHead`.
- Streaming downloads (`object.stream.pipe(res)` in assets, access and
issue attachments): they authorize before the stream starts. The memo
closes at the first write.
- `POST /board/chat/stream` and `POST
/companies/:companyId/skill-sources/discover` (NDJSON): POST, so no
memo.
- Live events, plugin API, runner and terminal WebSockets: they use
`server.on("upgrade")`, so they do not pass through Express middleware.
## What Changed
- New `server/src/services/authorization-read-scope.ts`:
`authorizationReadScope` (an AsyncLocalStorage scope for GET and HEAD)
and `readOncePerRequest`.
- `authorizationService.isInstanceAdmin` and `getActiveMembership` use
`readOncePerRequest`. The existing batch snapshot memo still comes
first.
- `issueReadSqlCondition` and `projectReadSqlCondition` use the same
`admin:<userId>` key for their instance admin check.
- For `source: "session"` board actors, the scope seeds `admin:<userId>`
with `req.actor.isInstanceAdmin`, which the session middleware read from
`instance_user_roles` in the same request.
- `createApp` mounts `authorizationReadScope` after the actor and Cloud
control middleware.
- `authorizationReadScope` closes the memo when the response starts or
ends, ignores entries older than 2 seconds, and drops rejected reads.
- New exported `readInstanceAdmin(db, userId)`: the one predicate for
`isInstanceAdmin`, `issueReadSqlCondition` and
`projectReadSqlCondition`.
- New test `authorization-request-reads.test.ts`: one role read and one
membership read in authorization per issue read, and a removed
membership gives 404 on the next request.
## Verification
- Red: without the scope, the issue read sends 3 `instance_user_roles`
statements (`expected ... to have a length of 2 but got 3`).
- Green: with the scope, it sends 1 role read and 2 membership reads.
The issue list plus the summaries request send 1 role read each.
- Revocation: delete the membership row, and the next GET returns 404.
HEAD returns 200, then 404 after the revoke.
- Stream: a GET that flushes headers, then re-reads the membership after
it is deleted mid-stream, sees the deletion.
- Background callback: a callback scheduled in a GET that runs after the
response and after a revoke reads again and sees the deletion.
- Age bound: an entry older than 2 seconds is read again. A rejected
read is not kept.
- Seed predicate: for users with an `instance_admin` row, another role
row, and no row, the session middleware value equals
`readInstanceAdmin`.
- Other actors: an agent key, a board key and a cloud tenant actor get
no seeded value.
- Response hooks: a later middleware that wraps `res.writeHead` and
`res.write` keeps working for JSON and streamed responses. This test
fails with a version that restores the original methods.
- Red: with the previous implementation, the stream, callback, age and
rejection tests fail.
- `pnpm exec vitest run` on `access-routes-hidden-floor`,
`access-routes-permissions-upgrade`, `access-service`,
`agent-auth-middleware`, `authorization-privacy-query`,
`authorization-request-reads`, `authorization-run-context`,
`authorization-service`: 8 files, 177 tests pass. Also
`inbox-mine-candidates`, `issue-read-filter-batching`,
`attention-summary-counts`, `issues-unread-inbox`,
`live-runs-batched-reads`, `privacy-production-review`,
`activity-routes`, `authorization-privacy-query`,
`muse-native-visibility`: 224 tests pass (one
`privacy-production-review` test failed once under machine load and
passed on two reruns).
- `pnpm --filter @paperclipai/server typecheck` passes.
## Risks
- This changes where authorization gets two facts inside one request, so
it needs a human review before merge.
- A GET route that changes a membership or an instance role and then
authorizes again before its response starts, within 2 seconds, would see
the earlier value. GET routes do not change memberships or roles. POST,
PATCH, PUT and DELETE requests do not use the memo.
- A board actor built by the session middleware still carries its own
connect-time `memberships` and `isInstanceAdmin` for the life of a
stream. That is the existing behavior, and this change does not affect
it.
- Nothing is cached across requests.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, test runs).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
canary/v2026.1115.0-canary.5
|
||
|
|
f33db357f7 |
fix(ui): let Request changes on a plan always take a revision note (#21721)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - In planning work, an agent saves a Plan document and asks the person to confirm it with a request_confirmation card > - The card offers "Request changes"; it opens a revision note only when the request sets rejectRequiresReason, allowDeclineReason or declineReasonPlaceholder > - Paperclip's own plan confirmations (runner plan approval) always set rejectRequiresReason, but an agent that writes its own confirmation may not > - Then "Request changes" rejects the plan at once and the person cannot say what should change > - This pull request makes plan confirmations always open the revision note; the note stays optional unless required > - The benefit is that a plan can always be sent back with a reason, and the QA plan journey can complete ## Linked Issues or Issue Description **What happened?** On the QA Cloud stack the plan-revise-accept journey failed. The agent saved a correct two-step Plan and a confirmation bound to the latest revision, with labels "Confirm" / "Request changes" and `rejectRequiresReason: false`. Clicking "Request changes" rejected the plan immediately; no revision note opened. **Steps to reproduce** 1. Create a request_confirmation that targets the issue's `plan` document without `rejectRequiresReason`. 2. Open the task chat and click "Request changes". 3. The plan is rejected with no note. **Expected behavior** "Request changes" opens the revision note so the person can describe the change, as for Paperclip's own plan approvals. ## What Changed - `TaskChatCompactInteractionCard`: `collectsRejectReason` is true for any plan confirmation (issue_document target with key `plan`). The note label shows "(optional)" when a reason is not required, and the reject button stays enabled with an empty note. ## Verification - Red/green: new `TaskChatInteractionCard.test.tsx` case "opens the revision note for a plan even when the request does not require a reason" fails on master (the click rejects) and passes with the change. - `ui` vitest for `src/components/task-chat` and `IssueThreadInteractionCard`: 68 files, 1119 tests pass. UI typecheck clean. - QA Cloud: the plan journey failed at this step in run 43671188287 (the only failing product step in a 22/24 run). It will be rerun after deploy. ## Risks - Low risk. Only plan confirmations change: one extra step (the note box) before a plan rejection. Non-plan confirmations are unchanged. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
a78be8b601 |
feat(native): send launch preparation and run digests to plugin task prepare (#21717)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Environment plugins can run a native Runner as a plugin task, and a warm session keeps that Runner between turns > - On each warm turn Core sends the plugin a `prepare` operation with the full launch requirements for the new run > - The requirements' `requirementsDigest` covers the run identity, and the run identity changes on every turn > - So a plugin cannot tell that only the run moved, and it must redo its whole preparation every warm turn > - This pull request gives the plugin two digests: one that stays the same while its preparation stays valid, and one that names the run > - The benefit is that a plugin can transfer only the run identity when nothing it reconciles changed ## Linked Issues or Issue Description **What existing behavior does this improve?** The `prepare` operation that Core sends to an environment plugin for a plugin Runner task, on cold starts and on warm attaches. **Subsystem affected** Native runtime, plugin Runner tasks, the Runner launch contract, and the plugin SDK environment task contract. **Current behavior** A warm attach sends the full requirements. The only digest available, `requirementsDigest`, includes `runId`, `turnId` and `itemId`, so it changes on every turn even when the launch is otherwise the same. **Proposed behavior** Core computes two digests for every `prepare`: - `preparationDigest`: the launch requirements an environment reconciles when it prepares a Runner: - the runner, lease and session identity; - provider, model and settings; - the prompt and instruction text; - the runtime context's instruction and skill bundles, working copy and connection instructions. It leaves out the `runId`, `turnId` and `itemId`, and the runtime context's MCP assignment and binding (with `aggregateDigest`, which hashes the assignment). MCP reaches the Runner through Core's tool authority on every run attachment, never through files the environment publishes. Core already continues a warm session across an MCP-only change (`nativeRuntimeContextsMatch` with tool refresh). The binding is also named per run (`native-mcp:<runId>`). So the digest stays the same across warm turns of one Runner, lease and session, including when the agent's MCP access changes. - `runDigest`: `preparationDigest` plus everything it leaves out: `runId`, `turnId`, `itemId`, and the MCP assignment and binding. It is unique to one run's launch. A plugin that advertises `launchDigestVersion: 1` in its task description gets both fields on `prepare`. **Reason and benefit** A plugin can compare `preparationDigest` with its saved preparation and transfer only the run identity when they match, instead of a full preparation. **Breaking changes** None. Plugins without `launchDigestVersion` get no new fields. ## What Changed - `packages/paperclip-runner/src/live/runner-provider-launch.ts`: add `runnerLaunchPreparationDigest` and `runnerLaunchRunDigest`. The preparation digest drops the MCP assignment and binding; the run digest takes the full requirements and includes them. Both use the same canonical JSON as `runnerLaunchRequirementsDigest`, with their own schema tags so the three digests never collide. Export them from the package and the server's vendored runner shim. - `packages/plugins/sdk/src/environment-tasks.ts`: add optional `preparationDigest` and `runDigest` to the `prepare` operation, and optional `launchDigestVersion: 1` to the task description. The capability gates both digests. - `server/src/services/native-runtime/plugin-runner-task.ts` and `native-session-executor.ts`: send both digests on `prepare` when the description advertises `launchDigestVersion: 1`. - `server/src/services/environment-task-runtime.ts`: recompute both digests from the requirements, and refuse a `prepare` whose digests are missing in part or do not match. - `tests/runner-e2e/source-sdk-public-inputs.json`: refresh the SDK public source pin. - Tests: - the preparation digest is stable across runs and changes when the runner, lease, session, instructions, model, approval policy or settings change; - with a runtime context, an MCP-only change (the run's binding, no binding, a changed access set) keeps the preparation digest and changes the run digest; - a change to the instruction bundle, instruction entry, a skill bundle, the skill set, the working copy or the connection instructions changes the preparation digest; - the run digest changes per run; - Core sends the digests only when advertised; - the runtime refuses forged or partial digests before it calls the plugin. ## Verification - `npx vitest run src/live/runner-launch-preparation.test.ts` in `packages/paperclip-runner`: 26 passed. - `npx vitest run src/services/native-runtime/plugin-runner-task.test.ts src/__tests__/environment-task-runtime.test.ts` in `server`: 160 passed. - `npx vitest run source-sdk-public-inputs.test.mjs source-sdk-evidence.test.ts source-sdk-public-type.test.ts` in `tests/runner-e2e`: 59 passed. - `pnpm typecheck` in `server` and in `packages/paperclip-runner`: clean. - Not run locally: the full suite. CI covers it. ## Risks Low risk. The digests are new optional fields behind a plugin capability version. Core computes them and checks them again before it calls the plugin, so a plugin can rely on them. They prove only that the launch requirements are unchanged. A plugin that skips work on a match must still check its own state, such as files it published into the sandbox. If the runtime context carries values that change per run, the preparation digest changes too, so a plugin does its full preparation. That is safe, but the plugin saves nothing. The MCP binding is the one such value Core sets per run today, and MCP is excluded entirely. A plugin that does publish MCP configuration into its sandbox must not rely on this digest for it. Values outside the requirements are not covered at all: the account identity and credentials, the GitHub broker, project IDs, and the plugin's own runtime revision. A plugin must keep checking those itself. A match lets a plugin skip publishing its launch configuration, not computing the descriptor: `prepare` must still return a launch whose `requirementsDigest` matches the current requirements, which carry the current MCP assignment and binding. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, code search). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |