mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:59:26 +01:00
da8cfda42e499e0c3a5206ef44b7d67407022585
4230
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
da8cfda42e |
perf(recovery): stop counting a run start as a change to every task of its agent (#21831)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The server runs issue recovery queues. The `stranded-assignments` queue checks assigned tasks that have no live run. One check costs about 22 database statements. > - #21729 made the 5-minute backstop re-check only tasks whose own rows changed. A change to the assignee agent's row counts as a change to every task of that agent. #21765 left out the agent's heartbeat writes. > - Execution start in `heartbeat.ts` writes `agents.status = 'running'` and `updated_at`, but not `last_heartbeat_at`. So each run start still makes every task of that agent "changed". The backstop window overlaps the previous one by 5 minutes, so each run start re-checks those tasks in two windows. > - On an almost idle hosted instance with one live run and a new run every few minutes, this gives bursts of about 1,450 checks for about 855 settled tasks, every few minutes. > - This pull request stops counting an agent row whose current status is `running` as a change to its tasks. > - The benefit is no re-checks of settled tasks beside a live run: 540 backstop checks in 30 simulated minutes drop to 6 (only the live task itself), and statements drop from 15,280 to 3,984. ## Linked Issues or Issue Description Refs #21729, #21765, #21795 No issue exists. This describes the problem with the fields of `.github/ISSUE_TEMPLATE/enhancement.yml`. **What existing behavior does this improve?** The changed-tasks backstop of the `stranded-assignments` and `assigned-work-invariant` recovery queues (`server/src/services/recovery/changed-tasks.ts`). **Subsystem affected** Server: issue recovery background work. **Current behavior** On staging at near idle (one live native run, about six new runs in 30 minutes), the stranded-check statement bundle runs in bursts of about 10 checks per second for about 2.5 minutes, then stops, then starts again about 4 minutes later. The tasks do not change. The cause: every run start writes the agent row without a heartbeat time. The backstop then loads every open task of that agent as "changed", in two consecutive windows. **Proposed behavior** An agent row counts as a change to its tasks only when the agent is not `running`. A pause, a resume, a hold or a configuration change of an idle agent still counts. A run's own task still gets its run hints and an immediate targeted check. **Reason and benefit** Recovery work becomes proportional to activity, not to the number of tasks held by an agent that just started a run. ## What Changed - `server/src/services/recovery/changed-tasks.ts`: the assignee-agent clause of `changedTaskCondition` now requires `ag.status <> 'running'`. A comment explains why run lifecycle writes are left out. - `server/src/__tests__/issue-recovery-live-run-agent-census.test.ts` (new): a staging-shaped census on the embedded test database with the real recovery workers and a fake clock. It seeds 800 settled in_review tasks with a succeeded latest run (3 workers, about 266 tasks each), 30 in_progress and 30 todo tasks without live runs, and one live native run. The run's agent gets the execution-start write, and the run publishes a run hint every 10 seconds for 30 simulated minutes. It also has three correctness guards: a task of the live agent whose own row changes without a hint is checked within the backstop bound; a resumed agent's task is checked within the backstop bound when the resume sends no hint; and the hourly full pass still checks every settled task while the run stays live. - `server/src/__tests__/issue-recovery-deadlines.test.ts`: a fast test that a run-start write is not a change and a pause during the run is a change. ## Verification Red on `origin/master` (fix reverted), green with the fix: | 30 simulated minutes beside one live run | before | after | | --- | --- | --- | | stranded checks, backstop wake | 540 | 6 | | stranded checks, hint wake (the live task's own run hints) | 178 | 174 | | checks of settled tasks | 534 | 0 | | database statements | 15,280 | 3,984 | | work signals beyond the run's own hints | 0 | 0 | - `npx vitest run src/__tests__/issue-recovery-live-run-agent-census.test.ts` (4 tests): the census fails on master with `expected 534 to be less than or equal to 50` and passes with the fix. The three guards pass. - `npx vitest run src/__tests__/issue-recovery-deadlines.test.ts -t "run start"`: fails on master, passes with the fix. - `npx vitest run src/__tests__/issue-recovery-deadlines.test.ts src/__tests__/issue-recovery-idle-backstop.test.ts src/__tests__/issue-recovery-activity-hints.test.ts src/__tests__/issue-recovery-signal-fanout.test.ts`: 31 passed, 1 skipped. - `tsc --noEmit -p server/tsconfig.json`: clean. ## Risks - Low risk. The change narrows one clause of the backstop query. No schema change. - If an agent is resumed and then starts a run in the same window, the resume no longer counts through the agent row. A resume already sends its own `pause_lifted` hint (`agent-lifecycle-driver.ts`), and the hourly full pass still covers every task. Before this change, a run finish already hid a resume the same way, because the finish writes a heartbeat time. - A configuration change of an agent while it runs is caught by its next idle write or by the hourly full pass, not by the 5-minute backstop. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code with tool use (shell, file edit, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
304e40fe9c |
fix(auth): report effective Cloud owner access to the board UI (#21817)
## Thinking Path > - Paperclip manages AI agents and the environments where they run. > - Cloud owners can receive instance-admin rights for their own dedicated instance through the existing owner setting. > - Authentication computes these rights for each request without storing an instance-admin role. > - The board UI reads `/api/cli-auth/me` to decide whether to show administration controls such as Tailscale. > - That response used only stored roles, hiding controls that the Cloud backend already authorized. > - This change reports the effective Cloud request rights so the UI agrees with the backend. ## Linked Issues or Issue Description **What happened?** A Cloud owner with `enableOwnerInstanceAdmin` enabled can successfully read Tailscale connections, while `/api/cli-auth/me` reports `isInstanceAdmin: false`. The Connectors page therefore hides Tailscale. The same stale summary can hide other instance administration controls. **Expected behavior** The access summary should reflect the Cloud request's effective privileges. Turning off owner elevation must remove those privileges on the next request. Session and board-key callers must continue to use the fresh stored-role snapshot. **Steps to reproduce** Sign in as a Cloud stack owner with owner instance-admin elevation enabled and no stored instance-admin role. Open Connectors and compare the access summary with a Tailscale connections request: the summary reports no admin access even though the backend permits the request. Searched existing PRs for `"cli-auth/me" cloud` and `tailscale access`; no duplicate fix was found. Related: [#21526](https://github.com/paperclipai/paperclip/pull/21526), which added device sign-in and exposed this hosted permission-summary gap. ## What Changed - `/api/cli-auth/me` uses the trusted Cloud actor's computed `isInstanceAdmin` value for `cloud_tenant` requests. - Session and board-key requests retain fresh stored-role checks. No roles are persisted and no authorization gates or owner settings change. - Regression tests cover Cloud elevation and de-elevation, non-Cloud stored access, and rejection of anonymous/agent callers. Tailscale documentation explains Cloud owner access. ## Verification - Reproduced the deployed mismatch in an authenticated Cloud owner browser session: the access summary reported false while the Tailscale list request succeeded. No feature connection or configuration was changed. - Both Cloud access regression cases failed before the fix. - `pnpm exec vitest run server/src/__tests__/cli-auth-routes.test.ts server/src/middleware/cloud-tenant-actor.test.ts ui/src/pages/apps/TailscaleConnector.test.tsx` passes: 88 tests. - `pnpm --filter @paperclipai/server typecheck` passes. - `pnpm verify --explain` exits 2 and selects broad CI. All 57 checks pass on final head `6044d5097396de95f49bcbb7ce7bbb1e445a80fa`; no checks remain pending. Greptile scored this head 5/5 with no unresolved comments, and the branch is conflict-free. - `git diff --check` and a local credential/PII scan pass. - The fix has not been deployed or accepted in a hosted browser yet. ## Risks The summary exposes administration controls to Cloud owners already authorized by the backend. Non-owner access and disabled elevation remain denied by the existing authentication policy. No migration, provider configuration, or deployment change. ## Model Used OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) through Codex in T3 Code, with xhigh reasoning. Context window size was not exposed. No subagents. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in-PR following the bug template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal ticket or instance details - [x] I have run tests locally and they pass - [x] I have added regression tests - [x] I have updated relevant documentation - [x] I have considered and documented risks - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no unresolved comments - [x] I will address actionable reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
f71aa135a0 |
fix(server): release sandbox leases for runs stopped before runner attach (#21788)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents in a `sandbox` environment run each task in a paid remote
sandbox that an environment lease tracks
> - When a run stops, the control plane must release that lease, and
terminating an agent waits until every lease of the agent is released
> - In a load test, a run stopped after its sandbox was ready but before
its runner started never released its lease, so the sandbox kept running
and the agent stayed in "terminating" forever
> - In the same test, terminating an agent whose failed run had a
scheduled same-run retry waited about 18 minutes, until the stale retry
time
> - This pull request makes lease release finish for every stop window,
records any release that must wait with a retry time, adds a hint-driven
safety net, and makes termination cancel a pending native retry at once
> - The benefit is that a stopped run never leaves a paid sandbox
running, and agent termination finishes in seconds without manual repair
## Linked Issues or Issue Description
**What happened?**
A load test with ten OpenCode agents in a `sandbox` environment stopped
its agents mid-run (pause, then terminate). Two defects appeared.
1. A run cancelled while it selected its provider pack (after the
sandbox was acquired and the workspace was prepared, before the runner
started) kept its environment lease `active` for more than 40 minutes.
The agent stayed in lifecycle `terminating`. The lifecycle worker
retried every pass and wrote `updatedAt` each time with no recorded
reason. Delete returned 409 "Complete termination before deleting the
agent".
2. A failed native run with a scheduled bootstrap retry kept its lease
until the stale retry time. Agent termination took about 18 minutes. The
run still showed "retry_scheduled" after the agent was terminated.
**Expected behavior**
A stopped run releases its sandbox lease within seconds. If release must
wait, the wait has a recorded reason and retry time, and a background
pass finishes it. Terminating an agent cancels a pending retry, releases
its lease, and reaches `terminated` promptly.
**Steps to reproduce**
1. Start a native run in a `sandbox` environment.
2. Stop it after the sandbox is acquired and the workspace is prepared,
before the runner starts.
3. Pause and then terminate the agent.
4. See that the lease stays `active` and the agent stays `terminating`.
**Paperclip version or commit**
Reproduced on `master` at
|
||
|
|
b4f71fc276 |
perf(recovery): stop re-checking settled workspace flags and snapshot ledger rows every pass (#21795)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The server runs all background recovery on one Node main thread, and
each database statement costs time on it
> - The ownership reconciliation pass settles in-progress records every
30 s while it has records to watch
> - Two of its kinds listed records that can never change again: dirty
remote workspaces whose writer reached runtime, and snapshot ledger rows
that are uploaded or still held by a stale writer's epoch
> - Each such record cost a transaction, a row lock and up to four reads
on every pass, forever, and kept the 30 s watch armed on an idle
instance
> - This pull request lists only the records that the settle step can
still act on, locks the writer run only when the settle will write, and
skips one stranded-task read that can never match a native run
> - The benefit is that recovery work follows activity, not history
## Linked Issues or Issue Description
**Describe the problem**
On a hosted instance with 7 native runs, the ownership pass ran about
7.6 + 3.2 + 3.5 + 2.8 statements per second over records that cannot
change, and took a row lock on `heartbeat_runs` for each stale dirty
workspace on every pass. At 1,000 agents this becomes thousands of
statements per second.
**Steps to reproduce**
Seed about 100 dirty remote workspaces whose writer run reached runtime
and ended, 50 pending ledger rows still held by those writers, and 100
uploaded ledger rows. Run the ownership reconciliation worker. The new
census test
`server/src/__tests__/recovery-settled-records-census.test.ts` does this
on the embedded database.
**Expected behavior**
A settled record costs zero statements on later passes. An idle instance
runs no SQL from this pass.
**Actual behavior**
On master, one pass costs 714 statements and 150 row locks. An idle half
hour costs 40,800 statements and 9,000 row locks (the settle watch
re-checks every 30 s).
Refs #21765 (the stranded-assignment hint scoping that this rebases on)
## What Changed
- `remote_workspace_unsaved_files` lists a dirty workspace only when its
writer run can still be a preparation-only writer: stage `preparing`, no
`runtime_mode_resolved_at`, no process, session or runner id, no
finalization row and no work-started event (or no writer at all). The
SQL lives with the settle step in
`remote-workspace-unsaved-files-settlement.ts`.
- The unsaved-files settle reads the writer run without a lock. It takes
`for update` only after every check passes, just before it writes, and
checks the locked row again.
- `remote_snapshot_ledger` lists only `pending` rows with no object key,
whose operation is a workspace save, and whose workspace epoch has moved
past the row's writer. Uploaded rows belong to snapshot retention, as
before.
- `terminalRunSettlingUntil` in `recovery/service.ts` skips its
`heartbeat_runs` read when the latest run already in memory is not a
legacy run of another server boot. `getLatestIssueRun` now also selects
`runtime_mode` and `controller_boot_id`. The live-execution check still
runs first.
- New census test with a staging-shaped fixture, a statement counter
grouped by statement shape and caller, and prompt-pickup checks. New
red/green test for the stranded-task lease read.
Census (embedded PostgreSQL, same fixture, base
|
||
|
|
c2e7df2dcc |
fix(workspaces): use project storage for a fresh retry after unowned task storage (#20756)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A project can use a storage provider plugin. That plugin prepares each task's git worktree > - A board Retry after a failed run can start a fresh attempt. The fresh attempt starts from the project source > - `preparePluginWorkspace` returns `null` when the task is still bound to a workspace that no storage provider owns. It does this also for a fresh attempt > - On a plugin task environment, the host then reserves an empty adapter-managed "Fresh task workspace" with no repository. The runner fails in it. The next Retry finds that empty workspace bound to the task, returns `null` again and reserves another empty workspace. The task can never start again > - This pull request lets a fresh attempt use the project storage provider even when the previous workspace is not provider-owned > - The benefit is that one Retry gives the task a normal provider worktree again ## Linked Issues or Issue Description **What happened?** Tasks moved off an older execution environment kept a binding to a workspace that no storage provider owned. A board Retry selected `fresh_isolated_workspace`. `preparePluginWorkspace` returned `null` because the bound workspace was not plugin-owned. The host fell back to `reserveFreshPluginTaskWorkspace`, which made an empty `adapter_managed` workspace (strategy `project_primary`, no repository, no recovery snapshots). The run failed in about one second with "Plugin-provided Runner execution operation unavailable; reconcile the same task before retrying". Each later Retry made another empty workspace, chained to the last one. **Steps to reproduce** 1. Use a project with a storage provider plugin that supports fresh workspaces. 2. Bind a task to a workspace that the provider does not own (for example an adapter-managed workspace). 3. Call `preparePluginWorkspace` for a fresh attempt with that workspace as the previous workspace. **Expected behavior** The storage provider prepares a new worktree for the fresh attempt. **Actual behavior** `preparePluginWorkspace` returns `null` and never calls the provider. ## What Changed - `server/src/services/plugin-workspaces.ts`: the early `return null` for a previous workspace that no provider owns no longer applies to a fresh attempt. The fresh attempt then uses the project's storage provider, as for a task with no bound workspace. - `server/src/__tests__/plugin-fresh-workspaces.integration.test.ts`: new test "prepares provider storage for a fresh retry that follows a workspace the provider does not own". ## Verification - The first commit adds only the test. CI on that commit (`d09ad235`, [failed job](https://github.com/paperclipai/paperclip/actions/runs/42976154239/job/130393285731)) shows only this test fail on the current code (`preparePluginWorkspace` returns `null`). - The second commit adds the fix. CI shows the test pass. - The other tests in the file keep the fresh-attempt guarantees: an older provider still fails closed, a changed binding revision still stops the call, and a lost reply still reconciles to the same operation. - I did not run the suite locally. The test needs embedded Postgres, and this environment does not install packages. CI runs it. ## Risks - Low to medium risk. A fresh attempt in a project with a storage provider now uses that provider even when the previous workspace was local or adapter-managed. Before, such an attempt used the fallback path. This matches the project's storage policy, and a fresh attempt does not reuse the previous workspace's files. - Projects without a storage provider are not affected: the function still returns `null` for them. - Non-fresh preparation is not affected. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), through Claude Code with tool use (shell, file edits). Standard reasoning mode. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: lucktastic <13757243+lucktastic@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
c6cdd57f7a |
perf(native-runtime): reuse a same-content image provider pack across builds (#21784)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A remote agent run needs the provider pack (Node, OpenCode and the Runner bridge) inside its sandbox before the provider can start > - The server reuses a pack that is installed in the sandbox image only when its full manifest equals the server's own manifest > - The manifest includes `runnerSourceRevision`, which is the commit that built the server, so an image pack fails this check on every later server build, also when every file in the pack is the same > - The run then uploads the whole pack again, which cost about 15 s per task in a recent load test > - This pull request accepts an image pack whose sealed content equals the expected pack and that differs only in build provenance, and caches verified archive hashes in the server process > - The benefit is that a sandbox image stays usable across server releases that do not change the pack, and the server does less hashing on its main thread ## Linked Issues or Issue Description Refs #21781 (merged), which names the failed check on the run spans. **What existing behavior does this improve?** Selection of the remote provider pack for a new sandbox (`provider_pack.select`). **Subsystem affected** Server: native runtime, remote provider pack verification and upload. **Current behavior** `provider_pack.verify_preinstalled` compares the complete manifest of the image pack with the server's manifest. Any difference fails, including `runnerSourceRevision` and `runnerInputsDigest`, which only record where the pack was built. The run then uploads the full pack. The server also hashes the same archive or pack directory again for every task. **Proposed behavior** When the run is not bound to a saved pack and the operator did not select a custom pack, the remote verification accepts an image pack when (1) its own digest seals its payload, and (2) its schema and payload equal the expected pack after the two provenance fields are removed. Every artifact, the `dist` tree, the candidate trees, the package versions, the Node version, the target platform and the OpenCode version are still checked in the sandbox, against the same values. The server checks the reported manifest again and selects it through the pack authority, as the compatibility path already does. An upload still happens for different content, a corrupted or partial pack, or no pack. The server also remembers a verified transport archive (by path, inode, size and times), so it does not hash it again for the next task. **Reason and benefit** In the harness (fake sandbox, 150 ms per command, 300 ms + 10 MiB/s per transfer, 24 MiB pack), the cross-build case goes from 4.1 s and 24 MiB uploaded to 1.1 s and 0 bytes. Other cases do not change. **Breaking changes** None. A saved or reattached session, and an operator-selected custom pack, still require the exact pack identity. ## What Changed - `remote-provider-pack-verification.ts`: new `acceptProvenance` option. The remote script accepts a sealed pack with the same content and prints its manifest. The server checks that manifest (`sameContentManifest`) and returns it. A forged or different report fails with `installed_manifest_invalid`. - `native-session-executor.ts` (`prepareRemoteProviderPackOnTarget`): staged and preinstalled verification use the option when `acceptProvenanceDifferences` is set. The function returns the selected manifest. The run caller sets the option only through `providerPackProvenanceReuseAllowed` (not retained, no explicit custom pack, selectable authority), and selects a different manifest through the pack authority. - `provider-pack-layers.ts`: `verifyArchive` skips the hash for an archive it already verified with the same file identity. - `remote-provider-pack.ts`: `providerPackProvenanceReuseAllowed` gate, with tests for each case. - `provider-pack-selection-harness.test.ts`: new harness. Uploads go through the workspace confinement check of the Daytona sync guard. It runs the real remote verification in local processes, with modelled sandbox latency and transfer speed, for five cases, before and after. ## Verification - `cd server && npx vitest run src/services/native-runtime/provider-pack-selection-harness.test.ts src/services/native-runtime/provider-pack-archive.test.ts src/services/native-runtime/remote-provider-pack-verification.test.ts src/services/native-runtime/remote-provider-pack.test.ts src/services/native-runtime/retained-provider-pack-source.test.ts src/__tests__/kimi-connection-probe.test.ts` passes. - `npx vitest run src/services/native-runtime/native-session-executor.test.ts src/services/native-runtime/native-artifact-preparation.test.ts src/services/native-runtime/remote-provider-pack-authority.test.ts` passes (1541 passed, 7 skipped). - Red/green: with the source changes reverted, the harness case `same_content_other_build` and the archive cache test fail. With the changes, they pass. - Harness output (load average about 9 on a shared machine; modelled time = wall time + modelled sandbox latency): | Case | Before | After | | --- | --- | --- | | Exact image pack | preinstalled, 1.1 s, 0 MiB | preinstalled, 1.0 s, 0 MiB | | Same content, other build | uploaded, 4.1 s, 24 MiB | preinstalled, 1.1 s, 0 MiB | | Different content | uploaded, 4.1 s, 24 MiB | uploaded, 4.0 s, 24 MiB | | Corrupted image pack | uploaded, 4.0 s, 24 MiB | uploaded, 4.1 s, 24 MiB | | No image pack | uploaded, 3.7 s, 24 MiB | uploaded, 3.8 s, 24 MiB | - The pack step runs no database statements. - `npx tsc --noEmit -p server/tsconfig.json` passes. ## Risks - This changes which provider pack the server accepts. It accepts a different manifest digest only when all content fields are equal and every file is hashed in the sandbox, and only for a run that is not bound to a saved pack. The selected manifest goes through the existing pack authority, so launch profiles and receipts bind the installed pack's own identity. - The archive hash cache trusts each archive file's identity (inode, size, times) on the server. The sandbox still checks the uploaded bundle digest and verifies every uploaded file, so a changed local file still fails closed. - This does not help when the image pack has other content (for example a different contract digest). See the hand-off notes for the image build. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
df9914dbd9 |
fix(native): let the outgoing server checkpoint its idle warm session while the incoming follow-up waits for it (#21791)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A deploy swaps servers with an overlap. A warm agent session stays with the outgoing server that ran its last turn. > - Since #21774, a follow-up that the incoming server claims during the overlap waits (`native_controller_recovery_waiting_for_owner`) and checks again every 5 s. > - About 1 s after the traffic switch (#21627), the outgoing server tries to save and close its idle warm session. The checkpoint fence refuses that save with `native_runner_checkpoint_owner_lost`. The task's current run is the parked follow-up. That run has its own finalization row under another controller boot, and the fence does not accept that kind of successor. > - The close retry from #21782 fails the same way each time. So the follow-up waits until the outgoing server exits, or until the predecessor's finish time plus the idle window plus 2 min. In the CHA-5 case that was about 225 s, and the worst case is about 425 s. Speed is a requirement. > - This pull request records which owner a parked follow-up waits on, and lets the fence accept the outgoing owner's final save when the follow-up is parked on exactly that owner and has not started. > - The benefit is that the outgoing server saves and closes the session on its first attempt, and the follow-up continues at its next 5 s check. ## Linked Issues or Issue Description Refs #21774, #21782, #21697, #21627, #21279 **What happened?** During a release-swap overlap, a warm follow-up was claimed by the incoming server while the outgoing server still owned the idle warm session. The follow-up waited correctly. The outgoing server's idle close failed with `native_runner_checkpoint_owner_lost` on every attempt. The wait ended only when the outgoing server exited, about 225 s after the follow-up was created. **Expected behavior** The outgoing server saves and closes the session while the follow-up waits. The follow-up sees the saved session and continues within one 5 s check. **Steps to reproduce** 1. Run a warm native session on a remote sandbox and let a run finish on server A. 2. Start server B with an overlap and switch traffic to it. 3. Post a follow-up comment so that B claims the follow-up while A is still alive. 4. A's idle close fails with `native_runner_checkpoint_owner_lost`, and the follow-up waits until A exits. **Paperclip version or commit** master `d14a3b88b1`. ## What Changed - `NativeControllerRecoveryWaitingError` carries the owner that the follow-up waits on: the predecessor run and its controller boot. `waitForLiveWarmSessionOwner` and the live-owner wait in `inspectRemoteWarmSessionRecovery` set it. - `deferNativeControllerRecovery` stores that owner in the finalization `failureDetail.waitingForOwner`. The executor passes it through. - `validateNativeRunnerCheckpointOwner` accepts one more successor kind: a parked follow-up waiting on this exact owner. All of these must be true, or the fence still refuses: - The checkpoint holds an accepted result and has no new work (unchanged rule). - The follow-up finalization is `retryable_failure` with `native_controller_recovery_waiting_for_owner`, `attempt = 0`, no result, and no lease owner (no claim is running it). - `waitingForOwner` names this owner's run and controller boot. - The follow-up run is a running native run for the same company and task, on the same native session as the owner's run. - The follow-up has no process fields, no own process-start events, and no checkpoint rows. - The fence locks the follow-up's finalization row (`for share`), so a resumption claim and the save cannot interleave. - Tests in `native-controller-recovery.integration.test.ts` (real Postgres): - The outgoing server saves and closes its idle warm session while the incoming follow-up waits, and the follow-up resumes within one recheck. - Negative cases: the save is still refused when the follow-up has its own process, its own process-start event, its own checkpoint, or a live resumption claim, or when it waits on a different owner boot or run. ## Verification - Red on master: with the `native-runner-checkpoint.ts` change removed, the scenario test fails with `native_runner_checkpoint_owner_lost` at the outgoing server's first close attempt. The follow-up keeps waiting. Green with the change. - The negative cases pass with and without the change. - Measured in the test scenario (warm idle window 600 s): - Before: the follow-up waits until the outgoing server exits, or about 720 s (finish + 600 s idle window + 120 s margin). - After: the outgoing save and close succeed on the first attempt. The follow-up is claimable and its wait resolves at `retryAt`, 4998 ms after it parked. - `vitest run` of `native-controller-recovery.integration.test.ts`, `native-runner-restart-recovery.integration.test.ts` and `native-harness-backup-storage.integration.test.ts`: 349 passed, 14 skipped. - `vitest run` of `native-session-executor.test.ts`, `remote-warm-session-recovery.test.ts`, `native-runner-checkpoint-save.test.ts`, `native-runner-checkpoint-retirement.test.ts` and `native-runner-checkpoint-authority.integration.test.ts`: 1566 passed, 7 skipped. - `tsc --noEmit` for `server` passes. - `pnpm verify` exits 2 (opaque setup dependency). Required CI supplies the broad coverage. - Not run: a release-swap wave. The fix is not yet measured on a live overlap. ## Risks - The fence is wider. It accepts only a final save with an accepted result and no new work, and only while the current run is parked on this exact owner and has not started. A follow-up that has a claim, a process, process-start events or checkpoint rows still causes a refusal. - A follow-up that a server parked before this change has no `waitingForOwner`. The fence refuses its owner's save as before. At its next recheck, the follow-up parks again with the owner recorded. - While a resumption claim runs the follow-up's recheck (a short window every 5 s), the save is refused. The close retry from #21782 then tries again. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
cec8c2b3f4 |
perf(recovery): name the run or task in run writers' recovery hints instead of the company (#21765)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The server runs issue recovery queues on one Node main thread. The `stranded-assignments` and `assigned-work-invariant` queues check assigned tasks that have no live run. One stranded check costs about 19 database statements. > - #21552 scoped run hints to the run's task graph. #21729 made the 5-minute backstop re-check only changed tasks. A hint with only a company id still makes the queue re-check every open task of the company, 25 per pass. > - Under load on staging (7 native runs), the stranded check ran about 6.7 times per second, and its statement bundle was more than a third of all statements. An activity fixture shows the cause: ordinary run writers send a company-only hint once per run, and one company hint costs about 850 checks. At 1,000 agents this work never stops. > - This pull request makes those writers name the run (or the task) they already know. A run that has no task now resolves to no tasks instead of the whole company. The changed-tasks backstop no longer counts an agent's own run heartbeat as a change to every task of that agent. > - The benefit: with 7 runs a minute plus one taskless run, stranded checks drop from 18.4 to 1.0 per second and stranded-check statements from 344 to 7 per second. Real company-wide changes (pause lifted, budget, company status, tree holds) keep the full company pass. ## Linked Issues or Issue Description Refs #21729, #21552, #21676, #21724 **What existing behavior does this improve?** The issue recovery task queues (`stranded-assignments`, `assigned-work-invariant`, `resolved-dependencies`, `stale-issue-locks`) and the writers that wake them. **Subsystem affected** Server: issue recovery background work and the run lifecycle writers that publish recovery hints. **Current behavior** Many run lifecycle writers know the run but publish a company-only hint, for example `notifyExecutionRunWork(tx, { companyId })` in the native finalizer and projection, `settleNativeTerminalWake`, the claimed-wake settlement, and an environment lease release of a run without a task. A queued run without a task publishes a company hint. A graph hint for a run without a task resolves to "whole company". A comment sends a company hint to `assigned-work-invariant` and `resolved-dependencies`. Each of these re-checks all open tasks of the company. Also, the changed-tasks backstop treats any change to the assignee agent's row as a change to all of that agent's tasks, and every run start and finish writes that row. **Proposed behavior** A writer that knows the run sends the run's graph hint. A writer that knows the task sends the task hint. A run without a task affects no task queue entry. An agent row change that is only the agent's own run heartbeat does not mark its other tasks changed. Company-wide facts keep the company hint. **Reason and benefit** The work is now proportional to activity, not to (activity x history). See Verification. ## What Changed - `server/src/services/issue-recovery-notifications.ts`: - `notifyIssueRecovery` with a task sends the task hint to all four task queues (before: only to `stranded-assignments` and `stale-issue-locks`; `assigned-work-invariant` and `resolved-dependencies` got the company hint). Only comments (`issue-comment-writes.ts`) call it with a task. - `resolveIssueGraph` returns `[]` when a run-only graph has no task. A missing task seed still returns `null` (whole company). - `server/src/services/recovery/work.ts`: a load for an empty graph returns no rows. The `assigned-work-invariant` queue emits the same `check` debug event as `stranded-assignments` (with `PAPERCLIP_DEBUG_WORK_SIGNALS=assigned-work-invariant`), with the load reason. - `server/src/services/recovery/changed-tasks.ts`: the assignee-agent part of the changed-tasks condition skips an agent whose last write was its run heartbeat (`last_heartbeat_at = updated_at`). A resume, pause or configuration write (no heartbeat stamp) still counts. - `server/src/services/heartbeat/queue.ts`: a queued run without a task gets its id before the insert and publishes a run hint, not a company hint. The daily-cap comment release publishes the run's graph hint. - `server/src/services/environments.ts`, `work-intent-outbox.ts`, `chat-inbound-wake-notice-notifications.ts`: a lease release passes its run id. `promoteDeferredWorkIntents` scopes `lock_released` with only a run to that run's graph. `notifyChatInboundWakeNotices` accepts an optional `runId` for its settlement hint. - Native and recovery run writers now pass the run id they already have to `notifyExecutionRunWork`, `writeExecutionControl` or `notifyExternalConversationSettlement`: `native-run-finalizer.ts` (3), `native-run-projection.ts` (2), `native-session-executor.ts` (3), `wake-request-claimed-settlement.ts`, `native-superseded-retry.ts`, `native-safe-replacement.ts`, `native-restart-recovery.ts` (2), `native-failed-workspace-recovery.ts`, `native-recovery-preparation-failure.ts`, `execution-control-reconciliation.ts`, `execution-recovery-resolution.ts` (2). These are the native equivalents of the legacy run paths in `heartbeat/queue.ts`, which already pass the run id. - Tests: - `server/src/__tests__/issue-recovery-activity-hints.test.ts` (new): staging-shaped company (780 in-review tasks with succeeded native runs, 40 in progress, 30 to-do, 50 agents) on embedded PostgreSQL, the real recovery, dispatch and retry-cancellation workers, a fake clock, and real process-adapter runs. Each run is woken by a user comment through `heartbeat.wakeup`, starts, calls back to post an agent comment and move the task to `in_review`, and finishes. One (or five) taskless runs a minute run beside them. It counts checks by queue and reason, task-queue signals by caller, and statements by owner. It asserts the workload (runs started and succeeded, agent comments, run events, zero agent write failures, at most 10% failed runs), then no company check and no company-only task-queue signal. In CI it runs the 7-runs-a-minute workload for 4 minutes (about 1 minute of real time). With `RECOVERY_ACTIVITY_REPORT=<file>` it runs both workloads at full length, attributes statements by stack, and writes the numbers below to that file. - `server/src/__tests__/issue-recovery-deadlines.test.ts`: pickup tests for each narrowed writer (see Verification). ### Publishers of company-only (or `*`) hints to the recovery task queues "Per" is how often the writer fires on a busy instance. "Known" says whether the run or task is known at the call site. Narrowed in this pull request: | Call site | Event | Per | Known | Now | | --- | --- | --- | --- | --- | | `heartbeat/queue.ts:5396` `enqueueWakeup` | queue a run without a task (timer, manual, chat without a task) | each taskless run | run (id generated) | run graph (no tasks) | | `environments.ts:1780` `releaseLease` -> `notifyChatInboundWakeNotices` | environment lease released | each run without a task | run | run graph | | `environments.ts:1792` `releaseLease` -> `promoteDeferredWorkIntents(lock_released)` | lease released | each run without a task | run | run graph | | `environments.ts:1934` `releaseLeasesForRun` | leases of a run released | each run without a task | run | run graph | | `issue-comment-writes.ts:12` (`assigned-work-invariant`, `resolved-dependencies` only) | comment | each comment | task | task | | `native-runtime/native-run-projection.ts:54` `settleNativeTerminalWake` | native run's wake claim settled | each native run finish | run | run graph | | `native-runtime/native-run-projection.ts:177` | native run status projected | each native run finish | run | run graph | | `native-runtime/native-run-finalizer.ts:408`, `:629`, `:2080` | native finalization terminal / failure / projection | each native run finish | run | run graph | | `native-runtime/native-session-executor.ts:13018`, `:13085`, `:13284` | native execution control deadline, cancelled retry, pre-result settlement | each native run end | run | run graph | | `recovery/wake-request-claimed-settlement.ts:67` | claimed wake settled after the run | each run whose claim is reconciled | run | run graph | | `heartbeat/queue.ts:590` | daily-cap comment key release | each run cancelled at the daily cap | run | run graph | | `native-runtime/native-superseded-retry.ts:42`, `native-safe-replacement.ts:613`, `native-restart-recovery.ts:1033`, `:1935`, `native-failed-workspace-recovery.ts:164`, `native-recovery-preparation-failure.ts:30`, `execution-control-reconciliation.ts:270`, `execution-recovery-resolution.ts:830`, `:965` | native recovery terminal writes | each recovered run | run | run graph | | `resolveIssueGraph` for any run-only hint of a run without a task (for example `setWakeupStatus`, `run-lifecycle.ts:1570`) | run lifecycle | each taskless run event | run | no tasks | Left with the company hint (company-wide fact, or the graph is not known): | Call site | Event | Per | Why kept | | --- | --- | --- | --- | | `budgets.ts:371`, `:386`, `:429`, `:437`, `:454` | budget policy changed, scope paused or resumed | operator or budget event | eligibility of every task can change | | `work-intent-outbox.ts:79` with `pause_lifted`, `tree_hold_released`, `scheduling_resumed`, `reconciler_backstop`, and `accounting_settled` with an active hard stop | hold release | operator event; `accounting_settled` once per run only with a hard-stop policy | can admit any task; guarded by "keeps the company pass for hold releases that can admit any task" | | `companies.ts:77`, `:350`, `:353`, `:361`, `:512`, `:522` | company created, paused, resumed, archived | operator | company-wide | | `agent-lifecycle.ts:65`, `:72`, `agent-lifecycle-driver.ts:29`, `agent-configuration.ts:130`, `:159`, `agent-deletion.ts:61` | agent paused, resumed, configured, deleted | operator | can change many tasks, and other agents' review paths | | `access.ts:570`, `:752`, `:912`, `:926`, `:960`, `:975`, `:1175`, `board-claim.ts:129`, `:142`, `middleware/auth.ts:706` | membership and authority changes | operator, first sign-in | company authority | | `chat-channels/endpoints.ts:461`, `identity-links.ts:323`, `resource-reconciliation.ts:290`, `slack/setup-services.ts:50`, `chat-github-management.ts:182`, `:352`, `chat-work-notifications.ts:50` | chat authority changes | operator | company authority | | `instance-settings.ts:666`, `:669`, `:680` (`*`) | instance setting changed | operator | instance-wide | | `issue-tree-control.ts:809`, `:889`, `:1236`, `:1339` | tree hold created or released | operator | many tasks | | `company-deletion.ts:83`, `company-portability.ts:6560` | company delete, import | rare | many tasks | | `issues.ts:8015` (blockers replaced), `:11165` (task created), `:12685`, `:12699` (task deleted), `:12991`, `:14618` | task graph changes | per task create or blocker edit | graph moves; #21552 kept these | | `documents.ts:615`, `:673`, `:757`, `:808`, `:860` | document delete and restore | rare | #21552 kept these | | `task-watchdogs.ts:854`, `:1841` | watchdog changes | rare | #21552 kept these | | `system-recovery.ts:17`, `:70`, `task-retry.ts:294`, `workspace-recovery-capture-policy.ts:59`, `workspace-recovery-stop-proof.ts:122` | system and workspace recovery | per recovery | not shown to stay in one graph | | `agent-conversations.ts:607`, `chat-channels/telegram/attachment-failure.ts:83`, `native-runtime/chat-attachment-reuse.ts:1754`, `native-runner-file-handoff.ts:1317` | conversation and attachment paths | per failure | not shown to stay in one graph | | `wake-queue/adapters/postgres.ts:376`, `:410`, `:555`, `:1169`, `routes/issues.ts:14552`, `issues.ts:12070`, `:12171`, `heartbeat/run-control.ts:1128`, `:1770`, `projects.ts:100`, `:1051`, `environments.ts:1617` (`notifyChatInboundWakeNotices` without a task) | saved notice and wake changes | per manual wake or project change | company admission state; not reached by run activity in the fixture | | `heartbeat/run-state.ts:1746`, `native-harness-backup-expiration.ts:356`, `:526`, `explicit-native-continuation.ts:705` | session goal clear, backup expiry, explicit continuation | rare | no run id at the call site | | `fast-responses.ts:201`, `:559`, `:672`, `:1053`, `chat-github-response-comments.ts:62`, `:229`, `:440`, `chat-github-reviews.ts:1063` (`notifyChatResponseAvailability` with a delivery only) | chat response availability | per chat response | no task at the call site; not measured here (chat paths are not in the fixture) | Per-run-event path: appending run events, saving native checkpoints, renewing leases and streaming deltas publish no recovery task-queue hint. `heartbeat-run-events.ts` reaches only live-run recovery queues (`native.process_*`, `turn.*` events), chat progress, response presentation, and `notifyNativeStatusReassessment` for control-plane evidence events only. Checkpoints and lease renewals reach `notifyLiveRunRecovery`, which does not include the task queues. A run's start and finish publish its graph hint from several writers (about 20 task-queue signals per legacy run in the fixture, for example `notifyChatRunMilestones` in `run-identity.ts` activation, `claimQueuedRun`, `run-completion.ts`, `setWakeupStatus`). They coalesce into about 3 checks of the run's own task per run. That is not one hint, but it is bounded by the run's graph. This pull request does not change it. ## Verification - Activity fixture (`issue-recovery-activity-hints.test.ts`), checks per second by queue and reason, and statements per second. Each workload minute is exactly 60 s of fake-clock time, and the rates use the elapsed fake-clock time. "Before" is this branch's test against the `origin/master` service files. Both versions did the same work (7 runs/min: 150 runs started, 143 to 145 succeeded, 133 to 135 agent comments, about 1,280 run events; 50 runs/min: 629 to 630 started, 620 to 622 succeeded, about 590 agent comments, about 5,550 run events). The test now asserts this workload and zero agent write failures before it accepts the counts. | | master | this branch | | --- | --- | --- | | 7 runs/min + 1 taskless run/min (600 s), stranded checks/s | 18.30 company + 0.07 hint | 0.91 hint + 0.06 backstop, 0 company | | 7 runs/min, statements/s in stranded checks | 343.9 | 7.2 | | 7 runs/min, all statements/s | 577.5 | 220.4 | | 7 runs/min, company-only task-queue signals | 710 | 0 | | 50 runs/min + 5 taskless runs/min (360 s), stranded checks/s | 16.86 company + 0.31 hint | 1.05 hint, 0 company | | 50 runs/min, statements/s in stranded checks | 314.0 | 7.4 | | 50 runs/min, all statements/s | 1,717.6 | 1,387.5 | | 50 runs/min, company-only task-queue signals | 2,864 | 0 | "Statements in stranded checks" counts statements whose stack includes `reconcileStrandedAssignedIssues`. The other statements are the runs themselves. Recovery load queries (graph lookups and hint loads) are 12.5 to 17.2/s at 7 runs and 69 to 72/s at 50 runs in both versions. `stale-issue-locks` saved deadlines check about 1 to 1.6 tasks/s in both versions. - Without the taskless runs, master already showed the backstop degrade under activity: with 7 runs a minute, 2.46 backstop checks/s, because every run writes its agent's row and marks all of that agent's tasks changed. With this branch: 0.27/s. - New tests in `issue-recovery-deadlines.test.ts`, red on `origin/master` service files and green here: - "re-checks a run's own tasks for its settlement, lease and dispatch writers, and nothing for a run without a task": for `notifyRunDispatch`, `notifyExecutionRunWork`, `notifyExternalConversationSettlement`, `notifyChatInboundWakeNotices` and `promoteDeferredWorkIntents(lock_released)` with a run id, the run's graph (task, parent, dependent) is checked within one pass and none of 300 backlog tasks is. For a run without a task, no task is checked. A comment checks only its task in `stranded-assignments` and `assigned-work-invariant`. - "counts an agent's own run heartbeat as no change to its other tasks, and a resume as a change". - "resolves a run without a task to no tasks and a missing task to the whole company". - Unchanged and passing: the #21729 pickup tests in `issue-recovery-idle-backstop.test.ts` (finished run, user comment, raised budget, resumed agent within a minute; missed hint within the backstop bound; outside change within the hourly pass), the #21552 tests in `issue-recovery-deadlines.test.ts` (including "keeps the company pass for hold releases that can admit any task" and "falls back to the company pass when a run affects more tasks than a hint names"). - Regression, embedded PostgreSQL: `issue-recovery-deadlines`, `issue-recovery-work-debug-attribution`, `issue-recovery-signal-fanout`, `issue-recovery-work-lifecycle` (42 tests); `native-run-projection-work`, `native-recovery-notice-work`, `muse-terminal-settlement`, `heartbeat-held-comment-wake`, `superseded-workspace-export-recovery`, `native-recovery-preparation-failure`, `upgrade-readiness-post-result`, `chat-wake-conversation-batch`, `environment-service`, `retained-native-session-cleanup-work`, `slack-conversation-lifecycle`, `workspace-recovery-dependency-finalize`, `native-failed-run-owner-release`, `chat-run-milestone-work` (462 tests). - `tsc --noEmit -p server/tsconfig.json` passes. ## Risks - **Recovery time bounds. The owner must see this.** No hint-driven pickup becomes slower for a task in the run's graph. Two missed-hint bounds change: - An agent change that sends no hint and is followed by a run heartbeat of the same agent inside the 5-minute window (for example a resume whose hint was lost, after which the agent ran another task) is now found by the hourly full pass, not the 5-minute changed-tasks pass. A normal resume sends `pause_lifted` and is checked at once (tested). - A task outside the run's graph whose recovery depended on the end of a taskless run, or on the native finish writers above, is now found by its own hint, the 5-minute changed-tasks pass (its own rows), or the hourly full pass. Before, any run finish re-checked the whole company. I found no recovery decision that waits on another run of the agent: concurrency waits are queued runs that dispatch resumes. - A run-only hint for a run row that does not exist resolves to no tasks (it was the whole company). A run row is not deleted while its tasks are open, except by company deletion, which sends the company hint. - Native finish writers now send run-scoped hints to the live-run recovery queues too (`notifyLiveRunRecovery`, `notifyLiveResourceRecovery`, attention retention). The legacy run paths already send these with the run id. - The fixture uses process-adapter runs, not native runs. The native writers are narrowed by code reading. Their effect on the hosted staging instance is not verified. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
10ddb26dbb |
test(server): accept dash's exit code for a sandbox-denied shell write (#21792)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Runner workspace bridge runs agent commands in a `bwrap` sandbox that binds only `/usr`, `/bin`, `/lib` and the task workspace > - A server test checks that a sandboxed command cannot read or write a file outside the workspace > - The test only runs where `/usr/bin/bwrap` exists. CI jobs that the gate sends to GitHub's `ubuntu-latest` now get the `ubuntu-26.04` image, which has `bwrap`, so the test runs on CI for the first time > - There it fails on every run, because it expects the write attempt to exit with 1, but Ubuntu's `/bin/sh` (dash) exits with 2 when a redirection fails > - This pull request checks that the write fails and that the file is unchanged, without depending on the shell's exit code > - The benefit is that the "General tests (server (7/12))" job passes again on the new image, and the test still proves the sandbox denial ## Linked Issues or Issue Description **What happened?** `server/src/services/native-runtime/runner-bridge-tools.test.ts` › `Runner workspace bridge > runs useful commands but denies files outside the workspace and injected credentials` fails in "ci / General tests (server (7/12))" on unrelated branches, on the first run and the retry: - PR #21765, head `98197c87e9` - branch `feat/run-loop-detector`, run 43732047843, head `b8c6e27429`, job 132866487949, image `ubuntu-26.04` version 20261111.196.1 ``` AssertionError: expected { exitCode: 2, signal: null, …(3) } to match object { exitCode: 1 } ❯ src/services/native-runtime/runner-bridge-tools.test.ts:180:19 ``` On jobs that ran on the RunsOn `ubuntu24` image there is no `/usr/bin/bwrap`, so the test is skipped (7 skipped instead of 6). **Expected behavior** The test passes wherever the command sandbox is available, and it fails if the sandboxed command can read or change the outside file. **Steps to reproduce** 1. In a Debian or Ubuntu container, run `/bin/sh -c 'printf stolen > "$1"' sh /nonexistent/dir/private.txt`. 2. dash prints `cannot create /nonexistent/dir/private.txt: Directory nonexistent` and exits with 2. bash exits with 1. `cat` of the same path exits with 1. 3. In the sandbox, the outside temporary directory is not bound, so the path does not exist. This is the same case. **Root cause** A stale test expectation, exposed by a CI image change. No commit on `master` changed the bridge, the sandbox or this test today (the last change to the test is from 2026-11-08). The test expected the shell write to exit with 1. That is bash behavior. On Ubuntu, `/bin/sh` is dash, which exits with 2 for a failed redirection. Until today no CI image had `bwrap`, so the test never ran on CI. **Production impact** None. The sandbox denies the access as intended: the `cat` step on the same run exits with 1 and its output does not contain the secret, and the write fails with `Directory nonexistent` because the outside directory is not visible in the sandbox. This is not a sandbox regression. ## What Changed - The write step now checks that the outside file still contains `PRIVATE` first, then that the command was not stopped and exited with a non-zero code. It no longer requires exit code 1 from the shell. - The read step and the success step are unchanged. ## Verification - Linux container (`node:24-bookworm-slim`, `/bin/sh -> dash`): the write command exits with 2 under dash and 1 under bash, and both are failures, as the test now accepts. - `npx vitest run src/services/native-runtime/runner-bridge-tools.test.ts` in `server/` on macOS: 11 passed, 6 skipped (the sandbox tests need Linux with `bwrap`). - I cannot run the sandbox test on this Mac. It runs in this pull request's CI only when the job lands on an `ubuntu-26.04` runner. ## Risks - Low risk. Only one test assertion changes. The test still fails if the write succeeds, because the file content check runs first and the exit code must be non-zero. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edit, GitHub CLI, Docker). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
3ee6049f71 |
perf(native-runtime): name why a preinstalled provider pack was not reused (#21781)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A remote agent run needs the provider pack (Node, OpenCode and the Runner bridge) inside its sandbox before the provider can start > - The server reuses a pack that is already installed in the sandbox image, or it uploads its own pack > - In a recent load test every task uploaded the pack, and this step was about 43% of the time to first token > - The compatibility check and the preinstalled-pack verification failed, but the run records did not say which check failed > - This pull request records the failed check on the run's performance spans > - The benefit is that the next run names the reason, so the image or the check can be fixed without guesswork ## Linked Issues or Issue Description **What existing behavior does this improve?** The run performance spans for provider pack selection. Today a failed `provider_pack.compatibility` or `provider_pack.verify_preinstalled` span only says `outcome: failed`. The code catches the error and drops it. **Subsystem affected** Server: native runtime, remote provider pack selection (`server/src/services/native-runtime`). **Current behavior** `provider_pack.select` records `reason: cached_packs_unverified` when it uploads. The compatibility check throws one generic message for about ten different conditions, and the caller swallows it. The remote verification script throws a generic `manifest mismatch` without the fields that differ. **Proposed behavior** Each failed check has a short, fixed name (for example `installation_unprotected`, `contract_digest`, `artifact:nodeCommand`, `manifest_mismatch:runnerSourceRevision`). The trace records this name as the `check` attribute on the failed span. `provider_pack.select` records the last preinstalled rejection as `check` when it uploads. The compatibility span records `target: image` or `target: workspace`. **Reason and benefit** Upload of the provider pack costs about 15 s per task on the test instance. To remove it, we must know which check rejects the image pack. The attribute holds only fixed check names and manifest field names, never paths or values. **Breaking changes** None. Error messages keep their existing text. The upload decision does not change. ## What Changed - `remote-provider-pack-verification.ts`: the compatibility check names the failed condition (`failureCheck`). The remote verification script prints a fixed `paperclip-pack-check=<name>` line, and for a manifest mismatch it names the differing manifest fields. Missing files now give a named check instead of a raw stack trace. - `native-run-trace.ts`: a failed `measure` records the error's `failureCheck` as the safe `check` attribute. - `remote-provider-pack.ts`: `provider_pack.select` records the last preinstalled rejection (or `no_preinstalled_pack`) when it uploads. - `native-session-executor.ts`: the `provider_pack.compatibility` span records `target`, the preinstalled-pack selection passes its rejection callback, and a failed link has the `link_failed` check. No other lines change. ## Verification - `cd server && npx vitest run src/services/native-runtime/native-run-trace.test.ts src/services/native-runtime/remote-provider-pack.test.ts src/services/native-runtime/remote-provider-pack-verification.test.ts src/__tests__/redaction.test.ts` passes (131 tests). - Red/green: with the source changes reverted and the new tests kept, 12 tests fail. With the changes, all pass. - The verification test runs the real remote script in a Node process: a provenance-only difference gives `manifest_mismatch:runnerSourceRevision`, a wrong artifact gives `artifact_digest:nodeCommand`, and a missing manifest gives `manifest_unreadable`. - `npx tsc --noEmit -p server/tsconfig.json` passes. ## Risks - Low risk. The change adds diagnostics only. The selection order, the checks themselves and the upload path do not change. - The `check` attribute is filtered by the existing safe-attribute list and a strict character pattern, so it cannot carry paths or secrets. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
d14a3b88b1 |
fix(native): release an idle warm session whose Runner is proven gone, and retry a failed idle close (#21782)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A warm agent session keeps its Runner alive in a sandbox between turns. The server closes idle sessions, and closes them again before it exits. > - When an idle close failed, the idle-timer path kept the session registered and never tried again, unless the sandbox lease was already inactive. > - The controller-restart path tried once more, but it accepted no proof that the session was already gone. It checked a destroyed sandbox only for a forced environment delete. > - In a release-swap test, one idle close failed with `runner_remote_process_identity_unavailable`. The session stayed registered until shutdown, the shutdown close failed again, and the sandbox was held until the host stopped it. > - This pull request treats a failed close as complete when the Runner is proven gone or the sandbox is stopped or destroyed, retries a failed idle close a bounded number of times, and logs which Runner check failed. > - The benefit is that a dead warm session is released, and the next failure names its cause. ## Linked Issues or Issue Description Refs #21627 **What happened?** In a zero-downtime release-swap test, a warm session's idle close failed 4 s after the server left new run starts. The failure was `runner_remote_process_identity_unavailable`. The idle-timer path kept the session registered with no retry. At SIGTERM, `closeIdleWarmNativeSessionsForRestart` tried again and failed the same way. The log showed "idle warm native sessions did not close" and the restart-chaos harness counted a warm-shutdown failure. The log did not say which Runner check failed, so the cause of the unreadable identity is not known. **Expected behavior** A close whose Runner is proven gone, or whose sandbox is stopped or destroyed, releases the session. A failed idle close is tried again a bounded number of times. The failure text names the Runner check stage and reason. **Steps to reproduce** 1. Keep a warm remote session idle. 2. Make its close fail with a `RemoteRunnerRecoveryError` (for example, `process_verification/process_not_running`). 3. Call `closeIdleWarmNativeSessionsForRestart()`. On master the result is `failed: 1`. ## What Changed - `server/src/services/native-runtime/native-session-executor.ts`: - New `warmCloseFailureProvesOwnerGone`. It uses existing proofs only: the close error's Runner diagnostic is `process_verification/process_not_running`, or a lease row for the sandbox has a termination receipt (`hasRemoteTerminationReceipt`) with state `destroyed`, or the warm run's own lease row has a receipt with state `stopped`. `pending_cleanup` and a lease without a receipt prove nothing. A lease read error counts as no proof. - `closeIdleWarmNativeSessions` (controller restart, environment delete, idle sleep) uses this proof for every failed close, not only for a forced environment delete. A released Runner detach failure is unchanged. - The idle-timer path uses the same proof. Its earlier status-only rule from #21504 (any lease status other than `active` or `retained`) is removed, because `pending_cleanup` is written before the provider stop. Without proof, it re-arms the timer and tries again, every 5 s, at most 3 times (`WARM_IDLE_CLOSE_RETRY_MS`, `WARM_IDLE_CLOSE_MAX_RETRIES`). The log line records the attempt number. - `describeWarmCloseFailure` appends the Runner check as `[stage/reason/lifecycle]` when the error carries one. - `server/src/services/native-runtime/native-session-executor.test.ts`: three tests. - "an idle warm session whose close fails with runner_remote_process_identity_unavailable and whose Runner is proven gone is released at controller restart" - "names the Runner check in a warm close failure and keeps an unproven owner registered at controller restart" - "retries a failed idle warm close a bounded number of times" - The #21504 idle-path test is now "forgets a warm owner whose idle close failed only once its sandbox has a stop or destroy receipt". A stop or destroy receipt releases the owner. `pending_cleanup`, `released` without a receipt, and `retained` keep it. - "reads the warm owner's own lease rows at controller restart" (a stop receipt for this run releases the owner; `pending_cleanup`, no receipt, or another run's receipt do not) ## Verification - `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts`: 1519 passed, 7 skipped. - Red on master (with only the two exported constants added so the file compiles): all three new tests fail. The first fails with `"failed": 1`. - `npx tsc --noEmit -p server`: no errors. ## Risks - A session is released without a checkpoint only when its Runner is proven gone or its sandbox is stopped or destroyed. The last saved checkpoint stays the recovery source, which is the same as the forced environment delete path today. - Environment delete and idle sleep now also accept the proof. Before, they accepted only a destroyed receipt during a forced delete. `pending_cleanup` is written before the provider stop, so it never counts as proof here. - The idle retry adds at most 3 more close attempts per failed session, 5 s apart. - A failed idle close whose sandbox stop is still pending now keeps its owner until a receipt lands or a successor retires it. Before, the status alone released it. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used in Claude Code with tool use (shell, file edit) and extended thinking. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
8f1d39da79 |
perf(server): cache the instance settings row per process (#21768)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - One server process runs every agent on one Node main thread, and
each database statement costs main-thread time
> - The `default` instance settings row is read on hot paths: live run
event delivery, run log redaction, the stranded-task recovery check,
runner tool calls, adapter routes and run preparation
> - A staging measurement with 7 live runs showed the full-row settings
read at 10.24 statements per second, and 3.5 to 5.4 per second when idle
> - The row changes only when an admin or a boot step writes it, so
almost every read returns the same value
> - This pull request adds a per-process read-through cache for that
row, with in-process invalidation on every write and a 2 second age
bound for changes made by another process
> - The benefit is at most one settings read per 2 seconds per server
process, whatever the load
## Linked Issues or Issue Description
**Is your feature request related to a problem?**
The full `instance_settings` row read (`select id, singleton_key,
default_environment_id, general, experimental ...`) runs at 10.24
statements per second on staging with 7 live runs, and 3.5 to 5.4 per
second at idle. It scales with run events and recovery checks, not with
settings changes.
**Describe the solution you would like**
Cache the row per process. Drop the cached row when a settings write
commits, in this process and, through the existing work-signal
notification, in the other process during a deploy overlap. Bound the
age of a cached row to 2 seconds, so any other change (a direct database
edit, a second process outside an overlap) takes effect within 2
seconds.
**Describe alternatives you have considered**
A database trigger with LISTEN/NOTIFY would also catch direct edits, but
it needs a migration, and work notifications cross processes only during
a deploy overlap today. The existing signal plus a 2 second bound keeps
the change small.
## What Changed
- New `server/src/services/instance-settings-cache.ts`: a read-through
cache keyed by the database client. Concurrent misses share one read. A
cached row is at most `INSTANCE_SETTINGS_CACHE_TTL_MS` (2000 ms) old,
measured from the start of the read. Every caller gets its own copy.
Reads through a transaction handle skip the cache.
- `instanceSettingsService` `get`, `getGeneral` and `getExperimental`
use the cache. Write paths (`update`, `updateGeneral`,
`updateExperimental`, and the feedback vote save) still read the row
fresh.
- Every settings write publishes an `instance-settings-changed` work
signal through the existing work-signal channel
(`@paperclipai/db/work-signals`). Inside a transaction it is the first
signal of the write, so its `pg_notify` payload goes out before the
other wakes of the same commit (for example `status-card-evaluation`).
This process drops the cached row when the write commits. During a
deploy overlap, when work notifications cross processes, the other
process drops its cached row on receipt, before a worker woken by the
same write reads the settings. A reconnect of the notification listener
also drops the row.
- `fillUnsetInstanceDefault` writes through `update({ db: tx })`, so its
signal is sent when its transaction commits.
- A read that starts before an invalidation never stores its result, so
an old value can not come back after a write.
- The test setup clears the cache before each test, because many tests
write the row directly.
Hot callers of the cached row:
- Live run event delivery: `live-run-event-redaction.ts`
(`censorUsernameInLogs`), once per delivery batch.
- Run output and run lifecycle redaction: `heartbeat.ts`
`getCurrentUserRedactionOptions`, used by `run-output.ts`,
`run-lifecycle.ts`, `run-completion.ts`.
- The stranded-task recovery check: `recovery/service.ts` reads
`getExperimental().enableAgentChat` once per checked conversation issue.
- Runner tool calls: `paperclip-runner-tool-authority.ts`
(`enableIsolatedWorkspaces`, `enableExternalObjects`).
- Adapter routes: `app.ts` (`enableNativeRunner`, `enableOpenAiDot`) per
request.
- Run preparation: `heartbeat.ts` (`instance_settings` preparation
step), `queued-run-line.ts`, `queued-run-dispatch-liveness.ts`,
`run-control.ts`.
- Issue and comment paths in `issues.ts` (`censorUsernameInLogs`,
`enableAgentChat`).
## Verification
- `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/instance-settings-cache.test.ts` (embedded PostgreSQL):
- 150 hot reads (`getGeneral`, `getExperimental`, `get` from new service
objects) after one warm read: 0 statements. Without the cache the same
test measures 150 statements (red run).
- 20 concurrent misses: 1 statement.
- `updateExperimental` and `updateGeneral` are visible at once to a new
read in the same process.
- `fillUnsetInstanceDefault` (transaction write of the default
environment) is visible at once after it returns.
- A change written by a second client (another process) becomes visible
within 2000 ms plus a 500 ms test margin.
- A caller that changes the returned object does not change the cached
row.
- A read through a transaction handle sees the transaction's uncommitted
write, two such reads issue 2 statements, and the cache does not keep
the write after rollback.
- A read that started before an invalidation is not stored.
- A received `instance-settings-changed` notification from another
process drops the cached row (next read issues 1 statement).
- `updateExperimental({ enableStatusCards: true })` sends
`instance-settings-changed` as its first notification, before
`status-card-evaluation`; `updateGeneral` sends it alone.
- The server tests that use instance settings or write the row directly
(230 files): see the run notes in the first comment.
## Risks
- A direct database edit, or a change from a second process outside a
deploy overlap, takes up to 2 seconds to apply in this process. Before
this change it applied at the next read. This includes access flags such
as `enableOwnerInstanceAdmin` (read per cloud-tenant request) and the
default environment.
- A change made through the settings service applies at once in this
process, and during a deploy overlap at once in the other process.
- A new code path that writes the `default` row must call
`announceInstanceSettingsChange`, or its change applies within 2
seconds.
- No schema change.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, test runs).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (no
user-facing setting or command changed)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
2b23119ed2 |
fix(native): keep a warm follow-up that inherited its predecessor's process identity waiting on the same run (#21774)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A deploy swaps servers with an overlap. A warm agent session stays with the outgoing server that ran its last turn. > - #21697 makes a follow-up wait while that outgoing server is still alive, instead of starting a second Runner over its session. > - Runtime selection copies the warm predecessor's process fields (`processPid`, `processGroupId`, `processStartedAt`) onto the follow-up and records `inheritedProcessIdentityRunId`. > - The wait records itself with `deferNativeControllerRecovery`. That function refused any run with process fields, so it refused every warm follow-up that the wait stopped. The run failed with `native_controller_recovery_wait_authority_changed`. > - This pull request lets the deferral accept an inherited identity, and clears those inherited fields in the same transaction so the usual resumption claim can pick the run up. > - The benefit is that a warm follow-up in a deploy overlap stays on the same run and does not fail. ## Linked Issues or Issue Description Refs #21697 **What happened?** In a zero-downtime release-swap test, a follow-up comment arrived 0.25 s after its predecessor finished on the outgoing server. The follow-up run started on the incoming server 1.09 s later. `waitForLiveWarmSessionOwner` found the outgoing server alive and asked to wait. `deferNativeControllerRecovery` then refused with `native_controller_recovery_wait_authority_changed`, because the run carried the process fields that runtime selection copied from its warm predecessor. The run failed after 2.7 s with `native_continuation_requires_reconciliation`. Held recovery started a fresh run. The task finished, but the run history shows one failed run. **Expected behavior** A warm follow-up whose only process identity is inherited from its predecessor waits on the same run. The resumption owner claims it at the deadline. A run whose own process has started is still refused. **Steps to reproduce** 1. Run a warm remote agent turn on server A. 2. Switch new run starts to server B while A still serves. 3. Post a follow-up within about 1 s of the turn's finish. 4. The follow-up fails with `native_controller_recovery_wait_authority_changed`. ## What Changed - `server/src/services/native-runtime/native-controller-recovery.ts` (`deferNativeControllerRecovery`): - Process fields refuse the deferral only when they are the run's own. Fields count as inherited when `runnerProfileJson.inheritedProcessIdentityRunId` is set and the run has no own `native.process_start_requested` or `native.process_identity_recorded` event. `run-cost-accounting.ts` uses the same proof. - With an inherited identity, the same transaction clears `processPid`, `processGroupId` and `processStartedAt`. It keeps `inheritedProcessIdentityRunId`. `claimNativeSessionResumptions` requires those fields to be null. - `server/src/services/native-runtime/native-controller-recovery.integration.test.ts` (real Postgres): - New: "keeps a warm follow-up that inherited its predecessor's process identity on the same run until the outgoing owner's deadline". - Two new cases in "does not publish an admission wait after authority changes": `own-process` (process fields, no marker) and `inherited-then-own-process` (marker plus an own process-start event). Both stay refused and keep their fields. ## Verification - `cd server && npx vitest run src/services/native-runtime/native-controller-recovery.integration.test.ts -t "inherited its predecessor|does not publish an admission wait|unstarted follow-up"`: 9 passed. - Red on master: the new test fails with `Error: native_controller_recovery_wait_authority_changed`. The own-process cases pass on master and with the change. - Green: the run is `retryable_failure` with `nextAttemptAt` at the deadline, its process fields are null, no claim happens before the deadline, and one claim happens at the deadline. ## Risks - The follow-up now waits instead of failing. The wait ends when the outgoing server saves a suspended checkpoint, when it exits, or at the predecessor's finish plus its idle window plus 2 minutes. In the observed case, the outgoing server's idle close could not save its checkpoint (`native_runner_checkpoint_owner_lost`), so the wait would end only when that server exits. See the follow-up note below. - An own process is still refused. The proof uses the run's own process-start events, the same proof as run cost accounting. **Follow-up note (not in this PR):** The outgoing server's checkpoint write checks the task's current run. A native successor that is parked in this wait is not one of the accepted successor kinds, and the idle close is not retried. So the wait does not end early from a checkpoint. In the observed timeline, the wait is about 225 s (until the outgoing server exited). The worst case with the default 5-minute idle window is about 425 s. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used in Claude Code with tool use (shell, file edit) and extended thinking. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
b9cf64f5bf |
test(server): check the agent row lock with FOR SHARE so lock checks cannot block each other (#21775)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Each master commit must pass Cloud readiness before its canary and image publish > - `agent-file-restore-locking.test.ts`, added in #21694, failed master Cloud readiness run 43712563988 on 2026-11-15, so that commit's canary did not publish > - The test reported the agent row as locked during the download, but its own sampler measured 0 ms of product lock time > - The cause is in the test: its two lock checks used `FOR UPDATE NOWAIT` on one pool and could block each other > - This pull request checks the lock with `FOR SHARE NOWAIT`, which still detects the product's `FOR UPDATE` but never conflicts with another check > - The benefit is that the test fails only when the product really holds the lock ## Linked Issues or Issue Description **What happened?** `downloads a stale projection without holding the agent row lock` failed in master Cloud readiness run 43712563988, attempt 2: `expected [{ objectKey, agentRowLocked: true }] to deeply equal [{ objectKey, agentRowLocked: false }]`. The test's metric line showed `lockedMs: 0`. **Steps to reproduce** 1. Make each sampler check hold its `FOR UPDATE NOWAIT` lock for 50 ms (to make the overlap certain). 2. Run `pnpm exec vitest run src/__tests__/agent-file-restore-locking.test.ts -t "stale projection"` in `server/`. 3. The test fails 3 of 3 times with `agentRowLocked: true` and `lockedMs: 0`, the same as CI. **Expected behavior** Only a lock held by the product makes the check report "locked". Refs #21694 ## What Changed - `agentRowLocked()` uses `FOR SHARE NOWAIT` instead of `FOR UPDATE NOWAIT`. The product takes `FOR UPDATE` on this row (`agent-file-store.ts`, `agent-instruction-revisions.ts`), which conflicts with `FOR SHARE`. Two `FOR SHARE` checks do not conflict. - New test: three concurrent checks all report unlocked, a `FOR UPDATE` held on another connection is reported locked, and the row reads unlocked again after release. ## Verification - Red: with the forced 50 ms overlap and the old `FOR UPDATE` probe, the test failed 3 of 3 runs with the CI error. - Green: with the same forced overlap and `FOR SHARE`, 4 of 4 tests passed in 3 of 3 runs. - Final file (no forced overlap): `pnpm exec vitest run src/__tests__/agent-file-restore-locking.test.ts` in `server/`: 5 of 5 tests passed in 5 of 5 runs. ## Risks - Low risk. Test-only change. The new test makes sure the check still detects a real `FOR UPDATE` lock. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
62033b9994 |
fix(qa): schedule recovered cleanup promptly and wait for task panels (#21660)
## Thinking Path > - Paperclip lets people manage AI agents and their saved work. > - QA journeys check browser controls and the cleanup of owned agent resources. > - A recovered workspace queues sandbox destruction after its files are saved. > - The cleanup queue treated that first attempt as a retry and waited five minutes. > - Agent termination correctly waited for destruction, so QA cleanup failed first. > - This PR runs that first cleanup attempt promptly and keeps the existing safety gates. > - It also waits for task-panel transitions before the browser acts again. ## Linked Issues or Issue Description **What happened?** The full scheduled QA smoke completed with 23 passed and 1 failed. Stop/redirect passed all eleven behavior checks but failed cleanup. The cancelled run had a saved workspace and a bound stop receipt before agent termination. Its sandbox was destroyed about five minutes later. The shared cleanup schedule applied the retry delay before the first destruction attempt. The browser regressions also found that the current panel helper can consume all four actions while an opener remains visible during panel mounting. **Expected behavior** A recovered workspace's first cleanup attempt runs on the next eligible worker pass. A failed attempt retains normal backoff. Agent termination still waits for completed resource cleanup. Task-panel navigation waits for the next controls before it acts again. **Steps to reproduce** 1. Queue recovered-workspace destruction after the cleanup worker has become idle. 2. Read the next cleanup deadline and advance one normal worker pass. Before this fix, the deadline is five minutes late and no teardown runs. 3. Open task document controls with a delayed side-panel transition. Before the panel fix, both collapsed-panel controls fail. Refs #19224 and #19292 for retaining stop proof during workspace recovery. This change keeps the full cleanup requirement for agent termination. Current master supplies the shared task-panel helper; this branch uses it and retains additional browser regressions. ## What Changed - Use one shared first-attempt predicate for the cleanup deadline, sweep selection and ownership report. It now includes recovered workspace cleanup bound to the same run. - Preserve worker spacing, in-flight claims, ownership and saved-file checks, retry delays and attempt limits. - Wait for panel controls after opening the panel. Wait for the selected tab or menu action to finish. - Add real PostgreSQL cleanup regressions, keep the Chromium controls, and document the cleanup timing. ## Verification Tested on Linux x86_64, Node v25.9.0. Local commit `7c8e0f12aa4b19aaf53669141ab3299ce76db963` and published head `892c5dd4b4638796a138400d8bbe30aee19c71dd` have the same tree, `d135a98bdff58e3320cdc83976ffe1cf47f48b66`. - Red: both new real-PostgreSQL cleanup tests failed on the old code. The deadline was five minutes late and the first normal sweep made zero teardown calls. - Green: 87 cleanup, ownership and lifecycle tests pass. The restored-work case stays unconfirmed until the real cleanup worker destroys the lease; failure still backs off and the worker becomes idle again. - Red after master integration: both delayed document-panel browser cases failed. Green: all 18 Chromium support checks pass. - 77 journey contract tests pass. - Product E2E typecheck passes with the documented low-memory path. The initial ordinary attempt was killed by the local memory limit and is not counted as a pass. - The only follow-up after the passing checks removes new code comments to follow the contribution guide. Runtime and test behavior are unchanged. Final-head [CI run 43714742904](https://github.com/paperclipai/paperclip/actions/runs/43714742904) completed successfully at this exact head. All 63 PR checks are terminal: 58 success, 2 neutral and 3 conditional skips; no failures or pending checks. Typecheck, build, tests and the final verify gate pass. Greptile and Paperclip Review both score this head 5/5 with no actionable findings; both historical inline threads are resolved. - Hosted QA recovery is still unverified. No extra model run, merge, deployment or QA setting change was made. ## Risks Recovered workspace destruction starts sooner. Existing producer, file-preservation, ownership, claim and provider checks still decide whether teardown is allowed. Failed attempts retain the existing cooldown. Agent termination and purge gates are unchanged. The panel helper waits for visible controls and still fails when controls or saved values are absent. The PostgreSQL integration uses a controlled provider boundary. It does not prove a new live Daytona or deployed QA pass. That requires the fixed revision on QA and a later normal scheduled smoke. ## Model Used OpenAI GPT-6-based Codex, with tool use and code execution. The host does not expose a more specific model ID or context-window size for this run. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub references) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
afca556728 |
fix(github): clarify configured review task completion (#21748)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - GitHub bots publish reviews and report the result of their assigned task. > - A GitHub check and a Paperclip task have separate completion rules. > - The current prompt explains how to wait, but gives little completion guidance. > - A bot can keep a complete review in review while it waits for CI or merge that its assignment does not require. > - This pull request makes the configured completion rules clear in GitHub prompts. > - The benefit is a finite review task with an accurate status and publication evidence. ## Linked Issues or Issue Description Related change: Refs #17318 keeps advisory review waits separate from human attention. This change adds completion guidance in the same GitHub prompt owner. **What happened?** A bot can publish a complete current-head 5/5 review and still wait for CI or merge. Manual request prompts also lack the task completion guidance used by automatic reviews. **Expected behavior** When the configured assignment allows completion at 5/5, a complete published review can finish Done. CI and merge are completion conditions only when the assignment requires them. Explicit tracking rules still apply. **Steps to reproduce** 1. Configure an agent to finish after a complete published current-head 5/5 review. 2. Start a GitHub review with CI results unavailable. 3. Inspect the generated prompt. It explains a response wait but does not explain this completion case or publication safeguards. 4. With this change, automatic, mention, and comment prompts carry the same explicit guidance. **Paperclip version or commit** Base commit: `83541e89ad3b90740ffc61a32c396dddb5834526`. **Deployment mode** Built from source. The prompt change applies to GitHub bot tasks. ## What Changed - Add shared guidance within the GitHub prompt module for configured completion at 5/5 and finite discussion replies. - Require a complete assessment, confirmed publication, and the current head in the 5/5 guidance. - Preserve explicit CI, merge, and below-5 tracking rules. Keep the shared status arbiter and check thresholds unchanged. - Add six prompt cases. State that CI is unavailable in the existing lifecycle completion case. - Document the configuration example and its limits. ## Verification - Six new prompt cases failed before the change. - `pnpm exec vitest run server/src/services/chat-github-review-policy.test.ts`: 40 passed. - `pnpm exec vitest run server/src/services/native-runtime/external-chat-wait.integration.test.ts -t 'settles a published'`: 9 passed. - `pnpm exec vitest run server/src/services/native-runtime/external-chat-wait.integration.test.ts -t 'inherits.*review|published.*guard|unknown publication|pending publication|changed head|inherited.*review'`: 5 passed. - `git diff --check` passed. - `pnpm verify` returned exit 2. The connector document requires broader CI coverage. - Local preflight installed dependencies and built shared and plugin SDK packages. It stopped because Cargo is unavailable. Full local typecheck, build, and test suites were not run. - [Greptile reviewed this commit](https://github.com/paperclipai/paperclip/pull/21748#issuecomment-6894002401) at 5/5 with no actionable findings. Its check passed for `36dd146cdc00b0f3d7f6fc2971f24166ef0f6bf5`. - [CI](https://github.com/paperclipai/paperclip/actions/runs/43711597984) passed for `36dd146cdc00b0f3d7f6fc2971f24166ef0f6bf5`. Typecheck, build, Rust/Runner, all server tests, and all eight E2E test groups passed. The final [CI gate](https://github.com/paperclipai/paperclip/actions/runs/43711597984/job/132808715297) passed at 12:17:58 UTC on 2026-11-15. - At 12:23 UTC, all 61 check records were complete: 57 passed, 1 was neutral, and 3 were skipped. Snyk also passed. No checks failed or remained pending. ## Risks This change gives prompt guidance. It does not enforce a score-to-status rule. Custom assignments can still require CI, merge, or later review. The focused tests check prompt content and existing lifecycle behavior; they do not prove every live agent or webhook journey. ## Model Used OpenAI GPT-6 through Codex, with reasoning, tool use, and code execution. This session does not expose the exact model variant or context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in this PR with the bug report fields - [x] I have not referenced internal or instance-local Paperclip issues or links - [x] My public branch name describes the change and contains no internal ticket id - [x] I have run focused tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
e6562755e1 |
fix(issues): cover a blocked issue whose own run is queued after its blockers finish (#21770)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A parent task is blocked on its subtasks. When the last subtask is done, Paperclip queues a run for the parent to continue. > - The board marks a blocked task "needs attention" when no live work covers it. The check looks at the blockers, then at each blocker's runs. > - When every blocker is done, the check returned "needs attention" at once. It did not look at the parent's own queued or running run. > - In a release-swap test, a parent showed "needs attention" for about 12 minutes while its own continuation run waited in the queue. Nobody had to do anything. The state cleared by itself. > - This pull request makes the check return "covered" when the blocked task has its own queued or running run. > - The benefit is that people see no false "needs attention" while Paperclip already continues the work. ## Linked Issues or Issue Description Refs #21731 **What happened?** In a zero-downtime release-swap test, a blocked parent task had two subtasks. Both subtasks were done. Paperclip queued the parent's "children completed" run, which waited about 12 minutes for a free agent slot. During that time, `blockerAttention` was `needs_attention` / `attention_required`, with 0 unresolved blockers and the parent itself as the terminal blocker. The board showed the parent in the blocked inbox. Then the run started, finished, and the parent became done. **Expected behavior** A blocked task whose own run is queued or running is covered by live work. It must not ask a person for attention. **Steps to reproduce** 1. Create a blocked parent task with one or more subtasks that block it. 2. Mark every subtask done. 3. Keep the parent's own run queued (for example, when the agent has no free slot). 4. Read the parent from the issue list. `blockerAttention.state` is `needs_attention`. ## What Changed - `server/src/services/issues.ts`: in `readIssueBlockerAttentionBatch`, when a blocked root has no unresolved blockers, the function now checks the active-issue set that it already built. If the root has its own queued or running run (or an active wake), it returns `covered` with `blockingTreeLive: true`. Otherwise it returns `needs_attention` as before. - The change adds no query. The active-issue set already includes the roots, through their `executionRunId` and their wake requests. - `server/src/__tests__/issue-blocker-attention.test.ts`: two tests. - "classifies a blocked parent whose children are all done as covered while its own children-completed run is queued" - "keeps a blocked issue with no unresolved blockers and no active run as needs_attention" ## Verification - `cd server && npx vitest run src/__tests__/issue-blocker-attention.test.ts`: 29 passed. - Red on master: the first new test fails with `"state": "needs_attention"`. Green with the change. - The second new test passes on master and with the change (negative case). ## Risks - Low risk. The change only affects a blocked task with zero unresolved blockers. That task now reads as covered while its own run is queued or running. When the run ends without a status change, the task reads as `needs_attention` again. - The covered result has no reason (`reason: null`), because no blocker covers it. The UI label is then "Blocked", not "waiting on 0 sub-tasks". ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used in Claude Code with tool use (shell, file edit) and extended thinking. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
5b750fdf25 |
feat(ui): add foldable sidebar subtasks with bounded refreshes (#21649)
## Thinking Path > - Paperclip helps people manage AI agents for work. > - The sidebar helps people return to recent tasks. > - Recent parents need foldable child tasks. > - Busy companies must not refresh every child group for unrelated work. > - Exact unread counts must preserve personal reads and task access. > - This pull request adds child groups and bounds their refresh work. > - People can scan subtasks without repeated background requests. ## Linked Issues or Issue Description **Subsystem affected** Task navigation in the standard and Classic sidebars. **Problem or motivation** Recent parent tasks did not show their subtasks. The first implementation refreshed every previously viewed group. One unrelated update caused 52 child-page requests and one summary request after viewing 50 groups. Exact child counts also repeated privacy and unread checks per child. **Proposed solution** Show direct subtasks with independent folding and ten-row pages. Refresh affected groups and summary batches. Stop offscreen reads. Bound child-request concurrency. Aggregate viewer facts once and preserve the existing inbox unread rule. **Roadmap alignment** This extends existing task navigation. The existing feature PR is reused for its performance repair. No new PR duplicates it. ## What Changed - Add shared parent groups to both sidebar layouts. Preserve Classic row actions and active-child visibility. - Keep unread state personal to each task. Folding and paging do not mark tasks read. Counts include readable children beyond ten. - Target live refreshes with authorized current and previous parent IDs. Reuse parent metadata from the narrow run-binding read. Resolve fallback lookups once per delivery batch. Keep the previous summary fallback for older servers. - Limit child requests to three at a time. Cancel active and queued reads when groups leave the viewport. - Discard extra cached pages on Show less and collapse. - Compute exact navigation counts in one SQL statement. Materialize readable candidate IDs once. Use indexed participation sources and conservative unread candidates before the shared inbox rule. - Skip recursive privacy walks for standalone open tasks without a project. Keep the recursive rule for all restricted or inherited shapes and all delegated principals. - Add request-count, cancellation, reparenting, read-receipt, inbox-parity and privacy regressions. Include a disposable PostgreSQL benchmark. - Update product and API documentation. Retain the production Storybook states. ## Verification - Published head: `4148861437c1b6b635479ad92509dbc71155a232`. Its tree matches local `04be9d66d481d9bcd1c4d6158acf063a1e65ac7e`: `a23533e93d3d155918f0254d296a7fd2f0c7c12f`. - Rebased on master `ca23330c69b1f2b25bdadfcbda68317db97deaef`. Remote publication preserves prior feature commits. - The request regression failed before the fix and passes after it. An unrelated event now sends zero child requests. Current servers also avoid the unrelated summary request. Old-server events keep one summary fallback. - A relevant update after Show less reads one page. Reparenting refreshes two affected groups. Offscreen reads stay at zero and resume once on return. Active and queued cancellation tests pass. - The same synthetic authenticated HTTP workload with 50,000 read children improved from 767 ms to 92.84 ms median. Five warm samples per case. The all-unread case improved from 463.85 to 351.65 ms. Untouched children improved from 208.01 to 131.09 ms. The old-receipt case improved from 799.70 to 245.80 ms. These local measurements are not a production capacity guarantee. - Focused PostgreSQL routes, inbox-parity, privacy and WebSocket tests passed: 38 tests. - All 325 affected UI tests passed. All 38 focused PostgreSQL, privacy and WebSocket tests passed. UI typecheck and low-memory server declaration checks passed. - All 41 Chromium Storybook checks passed. They cover 20 states in both themes and keyboard, folding, paging and unread behavior. They use production components with HTTP fixtures. - Both new review findings failed as regressions before their fixes and pass afterward. Explicit child reads include waiting external conversations. Deleting a child publishes a separate parent-authorized update without deleted-child content. Both review threads are resolved. - `pnpm verify --explain` returned exit 2 because product documentation requires broader CI. [The prior CI run](https://github.com/paperclipai/paperclip/actions/runs/43711489306) found type errors in the new test fixtures. The same errors failed locally. The fixture fix passes `node server/scripts/typecheck-layers.mjs --layer test-tree-0` and all 21 issue summary route tests. Production code is unchanged. That CI run then found a live-delivery read regression. The existing test failed locally with eight statements against its six-statement budget. A new PostgreSQL regression also found three parent lookups for two same-batch calls. Run delivery now reuses parent metadata from its narrow run-binding query. Fallback lookups resolve to one promise. Both regressions pass without increasing the budgets. All 74 targeted route, WebSocket and privacy tests pass. The affected below, above and test-tree-0 typecheck layers pass. [CI attempt 2](https://github.com/paperclipai/paperclip/actions/runs/43715541480/attempts/2) passes on the published head. All 62 check runs are terminal and clean: 57 succeeded, two were neutral and three were skipped. Typecheck, build, policy, all general and serialized server suites, E2E and the [final verify gate](https://github.com/paperclipai/paperclip/actions/runs/43715541480/job/132829783231) pass. The separate security status passes. Attempt 1's server shard 7/12 reached its 20-minute job limit without a test verdict. One retry of that job passed without a source change. [Greptile completed successfully on this head](https://github.com/paperclipai/paperclip/runs/132816163906) with a [5/5 score](https://github.com/paperclipai/paperclip/pull/21649#issuecomment-6883916606). All five review threads are resolved. The final CI gate is complete. - A full local Storybook build exceeded available memory while server typechecking ran. Browser verification uses the current-source development server. Rebuilding stale Runner and SDK declarations resolved the initial server check errors. The first browser attempt lacked Chromium; all checks passed after installation. - Reproduce the database measurements with `pnpm --filter @paperclipai/server exec tsx scripts/benchmark-sidebar-subtasks.ts`. - Extra local shard diagnosis ran 164 suites. Under this shell's `0077` creation mask, 157 suites passed and three had fixture permission or timing failures. Rerunning those three with `0022` resolved the permission failures. The remaining ten cases passed with one worker: two workspace cases and eight archive cases. The eight archive cases also passed on the pre-performance-fix revision, whose archive source is identical. The broad local shard is not claimed as one passing execution. The same-head CI retry supplies complete final CI coverage. - Nothing is merged or deployed. ## Risks - Exact counts still grow with the readable hierarchy and comment history. The change reduces repeated work; it does not cap or approximate counts. - A relevant summary batch can contain several recent parents. Older servers retain a broader summary fallback until upgraded. - Live delivery adds bounded, permission-checked parent lookups. Short delivery-window reuse limits repeated reads. - The authorization fast path applies only to open tasks with no privacy parent and no project. Restricted cases retain the shared predicate. - No migration is required. ## Model Used OpenAI Codex, based on GPT-6. Used code execution, native GitHub tools, PostgreSQL and Chromium verification. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either linked a public issue or described the issue in this PR - [x] I have not referenced internal or instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal issue id - [x] I have run focused tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation - [x] I have considered and documented risks - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open findings - [x] I will address all review comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
eb65fc0941 |
fix(server): turn agents' made-up-host Paperclip links into relative routes (#21762)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents post chat replies and task comments that link board pages and
attachments
> - Some models invent a hostname for these links, such as
`https://paperclip.local/COM/agents/<id>` or
`https://paperclip/api/attachments/<id>/content`
> - Those hosts do not exist, so the person clicks a broken link
> - The chat guidance already says never to invent a hostname, but a
model can still do it
> - This pull request rewrites such links to relative app routes before
an agent comment is saved, and adds the rule to the Paperclip skill
> - The benefit is that every board and attachment link an agent writes
opens, on whatever origin the reader uses
## Linked Issues or Issue Description
**What happened?**
Chat-eval cohort 16 (OpenCode with
`openrouter/deepseek/deepseek-v4.1-flash-1017`, master
|
||
|
|
56654adecd |
perf(server): reuse sandbox provider schemas for five seconds on the remote-run path (#21758)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agent runs can use remote sandboxes that a sandbox provider plugin supplies (for example Daytona) > - Each sandbox operation resolves the environment config, and that step reads the provider config schema to find the secret-ref fields > - On a hosted instance with 7 remote runs, this schema read was the most frequent statement: 17.95 per second, about 2.5 per second for each run > - The server runs all agents on one Node main thread, so each statement costs main-thread time; at 1,000 agents this read alone is about 2,500 statements per second > - This pull request reuses the retained schema for a short, bounded time and drops it at once when the plugin registry changes a plugin manifest > - The benefit is that a live remote run reads the schema about once per 5 seconds for each provider, not once per operation ## Linked Issues or Issue Description **What existing behavior does this improve?** Every plugin sandbox operation (exec, file sync in/out, runner ingress attach, capability snapshot, workspace realize, lease resume, lease release) calls `resolveEnvironmentDriverConfigForRuntime`. That call runs the `with selected_provider as materialized ...` statement in `resolvePluginSandboxProviderConfigSchema`, every time. **Subsystem affected** Server: plugin environment driver (`server/src/services/plugin-environment-driver.ts`) and the plugin registry manifest writes. **Current behavior** The schema read runs once per operation. A census test that drives the real runtime calls of one remote run counts 19 schema reads for 19 operations (see the table below). The hosted measurement was 17.95 executions per second with 7 runs. **Proposed behavior** The read runs at most once per 5 seconds for each provider and database client. A plugin install, upgrade or uninstall through the plugin registry in the same process takes effect on the next operation. **Reason and benefit** The schema is instance-wide plugin manifest data. It does not change during normal runs. Re-reading it per operation spends main-thread and database time for no new information. **Breaking changes** None. No API, schema or saved-state change. ## What Changed - `resolvePluginSandboxProviderConfigSchema` now returns the retained schema without a database read when the entry was checked less than `PROVIDER_SCHEMA_REUSE_MS` (5,000 ms) ago and the plugin manifest revision did not change. After that, it uses the existing cheap revision check, as before. - A lookup sequence number stops an older, slower lookup from replacing a newer retained entry. - A missing provider is still never retained, so a provider that becomes available is used on the next operation. - New module `plugin-manifest-revision.ts` holds an in-process manifest revision counter. It is a separate module so that the registry and the driver do not import each other. - `plugin-registry.ts`: `install` (both the new-row and the reinstall paths), `update` and `uninstall` announce a manifest change after the write completes. This is 5 changed lines plus 1 import line. Status-only writes do not announce, because the schema read does not use plugin status. - `plugin-loader.ts`: `installPlugin` runs `registry.install` inside an outer transaction. It now announces the manifest change again after that transaction settles, so a schema read made before the commit cannot be reused after it. This is 1 changed line plus 1 import line. - Statement census per operation, before and after. One remote run, one secret-ref field in the provider config, embedded test database, fake in-process plugin worker: | Operation | Count | Schema reads before | Schema reads after | All statements before | All statements after | | --- | --- | --- | --- | --- | --- | | Lease acquire | 1 | 1 | 1 | 26 | 26 | | Runner attach (ingress endpoint) | 1 | 2 | 0 | 11 | 9 | | Capability snapshot | 1 | 1 | 0 | 6 | 5 | | Exec | 10 | 10 | 0 | 50 | 40 | | File write (sync in) | 1 | 1 | 0 | 5 | 4 | | File read (sync out) | 1 | 1 | 0 | 5 | 4 | | Workspace realize | 1 | 1 | 0 | 5 | 4 | | Lease resume (reattach) | 1 | 1 | 0 | 6 | 5 | | Lease release | 1 | 1 | 0 | 18 | 17 | | **Total** | 18 | **19** | **1** | 132 | 114 | - Other reads that stay on each operation (reported, not changed in this pull request): each exec resolves each secret-ref field with 3 reads (secret, binding, version) and 1 `secret_access_events` insert. Runner attach and the capability snapshot also read the plugin row by id. These are part of the secret authorization and audit path, so they keep their own checks. ## Verification - `cd server && npx vitest run src/__tests__/plugin-sandbox-provider-schema-census.test.ts` (new). On master the census test fails with 19 schema reads. With this change it passes with 1. - The new tests also prove these points: - A registry upgrade that changes the secret-ref fields takes effect on the next call in the same process. - A direct manifest change (as from another server process) takes effect after the 5-second bound, and no read occurs inside the bound. - A registry uninstall takes effect at once. - A plugin reinstall through `pluginLoader.installPlugin` takes effect after its install transaction commits, even when a schema read ran inside the uncommitted transaction. This test failed before the post-commit announcement and passes with it. - A missing provider is not retained. The next call after install reads and uses it. - Two companies share the provider schema but each gets only its own secret values. A cross-company secret id still fails with "Secret must belong to same company". - A new database client (a process restart) starts cold, as before. - `cd server && npx vitest run src/__tests__/plugin-database.test.ts`: passed (plugin install transaction paths). - `cd server && npx vitest run src/__tests__/sandbox-provider-schema-reads.test.ts`: the existing tests now step the clock past the bound where they change the database directly. - `cd server && npx vitest run src/__tests__/environment-runtime.test.ts`: 210 passed. - `cd server && npx vitest run src/__tests__/bundled-plugin-registration.test.ts src/__tests__/external-object-refresh-work.test.ts src/__tests__/plugin-lifecycle-restart.test.ts src/__tests__/plugin-loader-error-retry.test.ts src/__tests__/plugin-environment-driver-sandbox-capabilities.test.ts src/__tests__/plugin-sandbox-provider-config-validation.test.ts src/__tests__/environment-probe.test.ts`: 82 passed, 1 skipped. - `cd server && pnpm typecheck`: passed. ## Risks - Staleness bound: a plugin manifest change made by a different server process (for example during an overlapping deploy) can take up to 5 seconds to apply here. During that time, a secret-ref field that an upgrade adds is not resolved yet, and a field that an upgrade removes is still resolved. The bound is short and it applies only to which config paths are secret refs. - The retained value is instance-wide plugin manifest data, keyed by database client and driver key. It contains no company data and no secret values. It does not take part in an authorization decision. Secret resolution, the environment binding check and the access audit still run on each operation for the calling company. - Memory stays bounded by the existing limits: 32 entries per database client, 64 KiB per schema. - `plugin-registry.ts` has a small edit in `install`, `update` and `uninstall`. Other open work in that file can cause a simple merge conflict. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edit, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Refs #17581 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
3244780bd0 |
fix(queue): keep a message a starting run's first turn took out of the queue (#21747)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A message posted while a run is starting goes into that run's first turn, and the task page shows it as sent > - The page shows it as sent only while the live queue lists it as a first-turn message > - When that queue closes, for example after a later Steer, the page has no lasting record of the delivery > - It then infers that the message is queued and shows it again with a Steer button > - This pull request reports each run's own first-turn receipt in the run list, and the page places those messages in that run > - The benefit is that a sent message stays sent ## Linked Issues or Issue Description Follow-up to #21661, found while I verified that change on staging. **What happened?** With a Codex runner, two messages were posted while the run was starting. The first turn took them and the agent acted on both. Then a third message was steered into the running turn. After the Steer, the first two messages came back into the queue with Steer buttons. The server queue was empty. **Expected behavior** Messages that a run's first turn took stay in the conversation as delivered. **Steps to reproduce** 1. Start a task with a Paperclip runner agent. 2. While the runner is starting, post one or two messages. 3. After the turn starts, post another message and click Steer. ## What Changed - `activityService.runsForIssue` returns `firstTurnCommentIds` from the run's `firstTurnQueuedInput` receipt. The field is set only when the receipt names this run, because a retry copies its predecessor's context. - `IssueDetail` adds those ids to the run's start inputs, so the messages are placed in that run and are not inferred as queued. - `RunForIssue` type gets the new optional field. ## Verification Red/green tests. Each failed before the fix and passes after it: - `server/src/__tests__/activity-service.test.ts`: "reports the queued messages a run's own first turn took, not a predecessor's". 33/33 pass. - `ui/src/pages/IssueDetail.test.tsx`: "keeps a message the running turn already took out of the queue after the live queue closes". The IssueDetail, queue helper, and task-chat suites pass (69 files, 1298 tests). UI typecheck passes. Staging, after #21661 deployed, Codex runner: two messages posted while the run started were taken into the first turn and acted on. No Interrupt, error banner, or composer wait was shown. A mid-turn Steer was accepted in seconds. This PR fixes the one defect that the check found. ## Risks - Low. One new read-only field on the run list. A run that took messages and then failed before its provider accepted the turn now shows those messages in that run. The next run still delivers them, and the queue still lists them while they wait. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
e7f31705da |
perf(release): stage the verified host release daemon instead of compiling it in publish_canary (#21555)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Every master push runs a canary, and only one canary runs at a time, so its duration sets how often merged work ships > - `publish_canary` takes about 30 min. 8 min of it is `pnpm build`, and 4 min 40 s of that is an uncached `cargo build --release` of `paperclip-runnerd` > - The release then replaces that binary with the exact-source daemons that `release_runner_binaries` already built and verified for all four targets > - This pull request stages the verified daemon for the publisher's own target during a release, instead of compiling it > - The benefit is about 4.5 min less per canary, with the same verified bytes published ## Linked Issues or Issue Description **What happened?** In canary run 43487884481 (2026-11-13), the `Publish canary` step took 28.6 min. Its `pnpm build` ran 19:42-19:50. The `packages/paperclip-runner build` took 5.3 min, and its `cargo build` reported `Finished release profile ... in 4m 40s`. Every other package waited for it. `release.sh` then runs `assemble-daemons`, which overwrites `dist/bin` with the four exact-source release daemons. **Expected behavior** A release does not compile a binary that it discards. The Runner build stamp stays valid, so the server build still reuses the Runner build. Refs #21554 (the other canary speed-up) ## What Changed - `packages/paperclip-runner/scripts/build-runner-binary.mjs` (new): when `PAPERCLIP_RELEASE_RUNNER_ARTIFACTS_DIR` is set, it checks the release daemons with the existing `releaseDaemonManifest` (source revision equal to `HEAD`, digest, architecture, build metadata) and stages the host target's daemon at `dist/bin` with the existing `stageRunnerBinary`. Otherwise it runs `pnpm run build:binary` as before. - The Runner `build` chain calls this script in place of `pnpm run build:binary`. `build:binary` itself and the image build are unchanged. - `stage-release-runner-binaries.mjs` exports `metadataContract`. - Tests: `stage-runner-binary.test.mjs` (staging per target, wrong source, bad digest, missing target, non-release fallback); `runner-image-identity.test.ts` step list. - `doc/RELEASING.md` describes the behavior. ## Verification - `node --test packages/paperclip-runner/scripts/stage-runner-binary.test.mjs`: 9 of 9 pass. - `server`: `vitest run src/__tests__/runner-image-identity.test.ts`: 42 of 42 pass. - With the real daemon artifacts of canary run 43487884481 (receipts pointed at this commit): `PAPERCLIP_RELEASE_RUNNER_ARTIFACTS_DIR=... pnpm run build` in `packages/paperclip-runner` finished in 15 s with no `cargo` line. The build stamp was recorded, and `server/scripts/prepare-runner-vendor.mjs` then reported `action:reuse,reason:inputs_unchanged`. The staged daemon runs. - After merge: the next canary's `packages/paperclip-runner build` log shows "Staged the verified exact-source linux-x64 release daemon" and no `Compiling` lines. ## Risks - Low. A release still fails closed: a daemon with the wrong source, digest, architecture or metadata stops the build, and `assemble-daemons` checks all four again afterwards. - Local, Docker and CI builds without the variable are unchanged. - On macOS, staging re-signs the binary, as it does for a compiled one. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
c17ef9249d |
fix(comments): keep saved authorization cards from blocking replies (#21744)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - People use task comments to ask for help with pending requests. > - Saving a comment also checks whether it replaces any pending requests. > - A stored request that fails today's payload schema can abort that comment write. > - The connection gateway could write such a request without validating its generated target. > - This change validates new authorization cards, reads older custom identities, and keeps malformed requests from blocking comments. > - Comments remain available while authorization and review gates retain their authority. ## Linked Issues or Issue Description **What happened?** A connection authorization card can contain a generated custom target key longer than 120 characters. The gateway writes the card directly. Later, the interaction reader rejects it. Both the task interaction list and comment submission then return a validation error. Other malformed saved requests can also abort a comment write. **Expected behavior** People can ask questions on a task while authorization is pending. New cards meet the shared schema. Older generated identities remain readable. A malformed request never becomes approval because a person posted a comment. **Steps to reproduce** 1. Use a personal connection with a long UID and no grant for the current user. 2. Invoke a tool. The gateway saves an authorization request with the long UID in its target key. 3. Open the task or post a comment. Reading the saved payload fails the key length limit. Related PR searches found no existing repair for this defect. This is a focused bug fix, with no roadmap feature overlap. ## What Changed - Validate generated connection authorization payloads before either insert or update. Use a stable SHA-256 target identity from the existing connection/user idempotency key. Bound its display label. - Read older confirmation custom keys and labels without the new-input length limits. Preserve their exact identities. Keep target types, URLs, document revisions, and resolver checks strict. New requests still use the strict shared schema. - During current and historical comment supersession, keep an invalid saved request pending, report its ID and schema locations, and continue with valid siblings. Database and other unexpected errors still propagate. - Keep this behavior in the gateway writer and interaction owner. The two existing source files exceed 1,000 lines; the additions extend their existing write/hydration boundaries rather than introduce another ownership path. ## Verification - Reproduced the deployed failure through the browser: comment POST and interaction GET both returned HTTP 400 for `target.key` exceeding 120 characters. The person's draft was restored. - Ran the new regressions before the repair: four expected failures and one short-identity control pass. Failures came from real payload parsing and real PostgreSQL comment writes, not fixture setup. - After repair, all 257 tests passed in the complete affected suites: `issue-thread-interactions-service.test.ts`, `tool-gateway.test.ts`, and `issue-thread-interactions.test.ts`. - Safety cases cover fresh and reused authorization cards, repeated missing-grant calls, refusing another user's credentials, two persisted follow-ups, strict new-input limits, wrong-company resolution, unsafe URLs, invalid document revisions, malformed question and confirmation payloads, and valid sibling supersession. - Local browser acceptance used an isolated, credential-free source instance on macOS arm64 with embedded PostgreSQL and Chrome. A card shaped like an older saved long-key authorization request remained visible. Two comments saved through Send, including a second after reload. The original comment persisted and the authorization request remained pending. No model execution was configured. - Browser acceptance was repeated on final commit `58b57dfe104b2ab421c7f24c89db4c29fcfda8ec` after rebase. The server health and ownership check confirmed that exact serving revision. Both comments persisted and the authorization stayed pending. Both owned instances were then stopped. All 257 affected tests passed again after rebase. The final fixture typing correction passed its six affected PostgreSQL cases. Greptile reviewed final head `58b57dfe104b2ab421c7f24c89db4c29fcfda8ec` at 5/5, with zero review threads or open findings. Required CI is green on that head: aggregate `ci / verify` and `ci / e2e` both passed. The repository CI run is https://github.com/paperclipai/paperclip/actions/runs/43710675824 . All 55 completed checks passed; four inapplicable checks were skipped. The non-required Superagent Security Scan remains in progress at 35%, with no annotations or reported findings. This external scan is still unverified; it was not retried or marked as passed. - `git diff --check` and the module boundary check pass. All seven local server typecheck layers pass on the final fixture revision. An earlier run found fixture typing errors; those were corrected and rerun successfully. Full repo tests, release qualification, and deployment were not run. ## Risks - Stored custom keys and labels may exceed current input limits. Only those length limits change for saved confirmation payloads; unsafe links and invalid document targets still fail validation. - A malformed request stays pending and still needs repair. It does not receive consent, disappear, or block a comment. Its read or resolution can still return a validation error. - This PR does not deploy the fix or recover a remote task. The original environment remains affected until rollout. ## Model Used OpenAI GPT-6-based Codex, with code editing, shell execution, browser tools, and PostgreSQL-backed tests. The host does not expose a more specific model version or context size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes (behavior and limits are recorded in this PR) - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
689d62661d |
fix(server): wake a delivery task unscoped when it cannot apply settled hints (#21686)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The delivery work coordinator wakes background tasks when a database work topic settles, and passes the settled scope hints to the task's `onSettled` > - The coordinator calls `onSettled(hints)` before `wake()` > - If `onSettled` throws on a hint that it does not recognise, the task is not woken. Only a generic listener warning is logged > - Hints now cross server processes (#21681), so a hint shape from another server version can arrive during a deploy overlap > - This pull request makes the coordinator log a warning, apply the settlement unscoped once, and always wake the task > - The benefit is that a hint that a task cannot read never makes the task miss committed work ## Linked Issues or Issue Description Follow-up to #21681 (merged). This branch is based on master. **What existing behavior does this improve?** How the delivery work coordinator handles an `onSettled` callback that throws. **Subsystem affected** `server/src/services/delivery-work-coordinator.ts`. **Current behavior** `subscribeDatabaseWork` calls the coordinator listener, which calls `task.onSettled?.(hints)` and then `wake()`. If `onSettled` throws, `wake()` does not run. The work-signal publisher catches the error and logs "Database work listener failed". The task stays asleep until a different event or deadline wakes it. **Proposed behavior** If `onSettled(hints)` throws, the coordinator logs a warning and calls `onSettled(undefined)` once, which is the unscoped settlement. If that also throws, it logs a second warning. In all cases it then wakes the task. **Reason and benefit** A settled event means committed work can be ready. A task must never miss it because a hint could not be applied. **Breaking changes** None. ## What Changed - `delivery-work-coordinator.ts`: a settled event goes through `applySettled(hints)`. It tries `onSettled(hints)`. If that throws and hints were given, it tries `onSettled(undefined)` one time. Each failure logs `Delivery settlement could not be applied for <queue>` with `process.emitWarning`. Then `wake()` runs. - Unscoped means `onSettled(undefined)`. Each registered task already defines what that means. For example, scoped deadline queues, dispatch work, muse maintenance and the revocation index mark `*` dirty. Counter-style tasks set their request flag. `chat-action-work` does nothing. The fallback is bounded: it calls `onSettled` at most twice per event, it does not recurse, and it does not schedule another settlement. So it cannot loop. - `work-signals.test.ts`: a new test pins one rule of #21681 for the local path and the remote path. An unhinted `signalDatabaseWork` in the same transaction as a hinted one gives a settlement that is scoped to the hinted set. The owner of `work-signals.ts` confirmed that this is deliberate. The repository does not allow code comments in pull requests, so this sentence and the test document it. ## Verification - `cd server && npx vitest run src/__tests__/delivery-work-coordinator.test.ts`: 31 pass. New tests: - An `onSettled` that throws on an unknown hint shape: the task gets `[hint]` and then the unscoped settlement, it is woken once, and one warning is logged. `onError` is not called. - An `onSettled` that always throws: the task gets 2 calls and is woken once, with 2 warnings. A later unscoped event gives 1 call, 1 wake and 1 warning. No more runs happen after 60 seconds, so there is no loop. - A normal `onSettled`: it gets the hints once, the task is woken once, and no warning is logged. - Red/green: with the coordinator from master, the 2 throwing tests fail. `onSettled` is called only with the hints, and there is no unscoped settlement and no wake. With this branch, all tests pass. - `cd packages/db && npx vitest run src/work-signals.test.ts -t alongside`: passes on real PostgreSQL. The local listener and the receiving process both get `["company-1"]`. ## Risks - Low risk. The change applies only when `onSettled` throws. Before, that path skipped the wake. Now it wakes with an unscoped settlement. For scoped deadline queues an unscoped settlement is a `*` reload. That is the existing safe fallback, and the reconstruction interval of the queue still throttles it. - A task whose `onSettled` throws on every event now runs on every settled event and logs a warning each time. Before, it did not run. The warning makes such a defect visible. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edit, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
83541e89ad |
perf(recovery): re-check only changed tasks between hourly full backstop passes (#21729)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The server runs issue recovery queues on one Node main thread. `stranded-assignments` and `assigned-work-invariant` find assigned tasks that have no live run and restart them. > - #21552 scoped run hints to the run's task graph and kept a company "backstop" pass to catch a missed hint. Any graph hint arms the backstop, and the backstop runs at most every 5 minutes after the last full pass. > - The backstop reloads and re-checks every open assigned task of the company. With any trickle of activity it runs every 5 minutes. On the hosted staging instance that is about 850 tasks every 5 minutes, about 2.8 checks per second at idle, close to the measured 3.2 per second. That is work proportional to history, not to activity. > - This pull request makes the 5-minute backstop re-check only tasks whose own rows changed since the previous look, with one set-based query. A full company pass runs at most once an hour, still armed only by activity. > - The benefit is no re-checks between hourly passes when nothing changed: idle statements drop from 59 per second to 5.5 per second in a staging-shaped fixture. A task whose own state changed without a hint is still recovered within 5 minutes. ## Linked Issues or Issue Description No issue exists. This describes the problem with the fields of `.github/ISSUE_TEMPLATE/enhancement.yml`. **What existing behavior does this improve?** The missed-hint backstop of the `stranded-assignments` and `assigned-work-invariant` recovery queues (`server/src/services/recovery/work.ts`). **Subsystem affected** Server: issue recovery background work. **Current behavior** Each graph hint arms a `{companyId, backstop: true}` scope. When it opens (5 minutes after the last full pass), the queue loads every open assigned task of the company and checks each one again: about 22 statements per task for the stranded check. With a hint at least every 5 minutes, this repeats every 5 minutes, whatever changed. **Proposed behavior** Between full passes, the backstop loads only the tasks whose own rows changed since the previous look (minus a 5-minute overlap for late commits): the task row, its runs, its recovery actions, its wake requests, its thread interactions, its comments, or its assignee agent. A full company pass runs when the last one is an hour old. A real change still sends its own hint and gets an immediate targeted check. **Reason and benefit** On staging at idle, the stranded check runs about 3.2 times per second with zero runs. A fixture with staging's population shape attributes all of it to the backstop. ## What Changed - `server/src/services/recovery/work.ts`: - `fullPasses()` keeps the time of the last full pass and of the last changed-tasks pass per company. The backstop gate is 5 minutes after the later of the two. - A backstop scan decides on its first page: a full load when the last full pass is at least `ISSUE_RECOVERY_FULL_BACKSTOP_MS` (1 hour) old, otherwise a changed-tasks load since the previous look minus `ISSUE_RECOVERY_BACKSTOP_MS`. All pages of one scan use the same decision. - Each scan records its start time on its first page. A completed scan advances the saved full or changed time only to that start, so a change behind the cursor of a slow scan falls in the next window. - The stranded queue adds `changedTaskCondition(since)` to its load query. The assigned-work queue passes `changedSince` to `listSuspects`. Both return `keepUnseen: true` for a changed-tasks load. - A new `fullBackstopMs` option. When a caller sets only `backstopMs` (tests), every backstop stays a full pass, as before. - With `PAPERCLIP_DEBUG_WORK_SIGNALS=stranded-assignments`, each `check` event now names why the task was loaded: `deadline`, `reload` (unscoped reconstruction), `backstop`, `company` (company-wide hint) or `hint` (task or graph hint). - `server/src/services/recovery/changed-tasks.ts` (new): the set-based "changed since" condition. Each part uses an existing index: `issues (company_id, updated_at)`, runs by `(company_id, context issueId)`, wakes by `(company_id, payload issueId)`, interactions and comments by `(company_id, issue_id)`. - `server/src/services/scoped-deadline-work.ts`: a load result may set `keepUnseen`. Then a completed scan does not drop the pending entries it did not load, so their saved deadlines stay. - `server/src/services/recovery/assigned-work-sweep.ts`: `listSuspects` accepts `changedSince`. - `server/src/services/issue-recovery-notifications.ts`: `ISSUE_RECOVERY_FULL_BACKSTOP_MS = 60 * 60_000`. - Tests: - `server/src/__tests__/issue-recovery-idle-backstop.test.ts` (new): the staging-shaped fixture, see Verification. - The staging-shaped population helper and the fixture's statement meter come from #21724, now merged into this branch. - `server/src/__tests__/issue-recovery-work-debug-attribution.test.ts`: the check event now names its load reason (`hint` or, when the backstop loaded the task last, `backstop`) instead of `load`. - `server/src/__tests__/issue-recovery-deadlines.test.ts`: the backlog helper writes its rows with the test's fake clock, as production writes use the server clock. Without it, rows stamped with the database clock look older than the fake-clock window. ## Verification - Fixture (`issue-recovery-idle-backstop.test.ts`, embedded PostgreSQL, the real recovery, dispatch and retry-cancellation workers, fake clock, statement meter, `PAPERCLIP_DEBUG_WORK_SIGNALS=stranded-assignments:1`): one company, 50 agents, 780 in-review tasks with succeeded native runs and no execution state, 40 in-progress and 30 to-do tasks of a paused agent, 12 tasks with an active agent-owned recovery action, 3 tasks of a terminated agent. After boot, one unrelated graph hint per minute stands in for idle activity. - Wake reasons on master over 30 idle minutes with that trickle: 5,100 stranded checks, all `backstop`; 0 from hints, 0 from company hints, 0 from deadlines. That is 2.83 checks per second and 106,632 statements (59.2 per second). With no hint at all after boot, master still made one backstop pass (850 checks) in 30 minutes, armed by boot activity. - This branch, same fixture: | | master | this branch | | --- | --- | --- | | Stranded checks, first 30 idle minutes | 5,100 | 0 | | Statements, first 30 idle minutes | 106,632 | 1,044 | | Stranded checks, one idle hour (one full pass) | about 10,200 | 850 | | Statements per second over that hour | 59.2 | 5.5 | - Missed hint: a task inserted with no hint is checked and gets a run within 5 minutes plus one pass (`ISSUE_RECOVERY_BACKSTOP_MS + 2 min` asserted). - A change outside the task's own rows (its agent resumed by a raw update that sets no `updated_at`, with no hint) is not seen by the 5-minute pass. It is checked and gets a run after more than 10 minutes and within 62 minutes, through the hourly full pass. - Prompt pickup with the backstop closed: a finished run (`notifyNativeStatusReassessment`), a user comment, a hold release (`pause_lifted`, which a budget raise and an agent resume send) and an agent resume each get a check of the affected task within 60 seconds. The company-wide events re-check the whole company at 25 tasks per pass, as before. - Two regression tests in `issue-recovery-deadlines.test.ts`, red on |
||
|
|
07d5a6ee17 |
perf(db): carry scope hints in cross-process work notifications (#21681)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Server processes share durable queues and wake each other with a
commit-bound PostgreSQL NOTIFY
> - The NOTIFY payload carries only `{version, origin, topic}`, so the
receiver gets an unscoped "settled" event
> - In `scoped-deadline-work.ts` an unscoped settle becomes `*`, which
reloads every candidate of that queue on the whole instance
> - On a large instance one cross-process notification can re-check
about 850 open tasks, at about 22 statements each
> - This blocks more than one server process per stack, and it costs
work during every deploy overlap
> - This pull request puts the topic's hints in the NOTIFY payload, so
the receiver reloads only the hinted scopes
> - The benefit is that a scoped commit in one process costs the same
scoped reload in every other process
## Linked Issues or Issue Description
**What existing behavior does this improve?**
Cross-process work notifications between server processes on one
database.
**Subsystem affected**
`packages/db` work signals (`work-signals.ts`) and their consumers, such
as `server/src/services/scoped-deadline-work.ts`.
**Current behavior**
`receiveDatabaseWorkNotification` publishes "settled" with no hints. A
scoped deadline queue then reloads all of its candidates, throttled only
by its reconstruction interval.
**Proposed behavior**
The NOTIFY payload carries the deduplicated hints of its topic. The
receiving process publishes "settled" with the same hints that a local
commit publishes. Overflow and malformed input fall back to the old
unscoped behavior.
**Reason and benefit**
A scoped commit no longer causes an instance-wide reload in other
processes. This is necessary before a stack runs more than one server
process.
**Breaking changes**
None. Old and new processes interoperate in both directions (see Risks).
## What Changed
- `packages/db/src/work-signals.ts`: the per-level pending notifications
are now a map of topic to `{ hints, unscoped }`. Savepoint merge and
rollback keep the same rules as before.
- The hints follow the local "settled" semantics: the union of the hints
from `signalDatabaseWork` and `publishDatabaseWorkSettled`. An unhinted
`publishDatabaseWorkSettled`, `notifyDatabaseWorkOnCommit`, or a topic
with no hints sends the unscoped payload.
- A hinted payload is `{version: 1, origin, topic, hints: [...]}`. The
field is additive and the version stays 1.
- Each payload is at most 1,024 bytes
(`DATABASE_WORK_NOTIFICATION_MAX_BYTES`), because current receivers drop
any payload longer than 1,024 characters before they parse it. Many
hints split into several payloads. The payloads of one topic are at most
7,000 bytes in total (`DATABASE_WORK_TOPIC_NOTIFICATIONS_MAX_BYTES`). If
a topic exceeds this cap, or one hint alone does not fit, the topic
sends one unscoped payload.
- All payloads still go in the one batched `select pg_notify(...) from
jsonb_array_elements_text(...)` statement at commit. There are no added
statements or awaits on the commit path.
- `receiveDatabaseWorkNotification` accepts payloads up to 8,000
characters (the PostgreSQL limit). It keeps the drop of own-origin
payloads and of payloads without a usable origin or topic. A missing,
malformed, or empty hint list, or a version other than 1, gives an
unscoped wake. It never throws.
- The `remote_notification` debug event now logs the received hints
(`hints`: a JSON array, or `null` when unscoped).
- Tests: new payload, compatibility and fallback tests in
`packages/db/src/work-signals.test.ts`, and a new two-process test on
real PostgreSQL in
`server/src/__tests__/scoped-deadline-work-notifications.test.ts`.
## Verification
- `cd server && npx vitest run
src/__tests__/scoped-deadline-work-notifications.test.ts`: two pools on
one embedded PostgreSQL database. Process A commits work. Process B runs
a real delivery coordinator with `registerScopedDeadlineWork` over 800
candidate rows in 20 companies.
- Master (red): a commit hinted with one company makes B load `*`: 1
load, 801 rows.
- This branch (green): B loads only `company-7`: 1 load, 41 rows. A
commit with two hints loads only those two companies (80 rows). An
unscoped commit still loads all 801 rows.
- `cd packages/db && npx vitest run src/work-signals.test.ts` (32 tests,
real PostgreSQL). New tests:
- deduplicated hints per topic, including a released savepoint and a
rolled-back savepoint
- split into payloads of at most 1,024 bytes, at most 7,000 bytes in
total, and the union of hints is complete
- overflow and one oversized hint fall back to one unscoped payload
- malformed hint lists and unknown versions wake unscoped; unusable
payloads do not wake; own-origin payloads are dropped; nothing throws
- an old-format payload `{version, origin, topic}` gives an unscoped
wake
- every new payload passes a copy of the receiver from master, which
reads it as an unscoped wake for the correct topic
- the debug event logs the received hints
- With master's `work-signals.ts`, 5 of these tests fail (red). With
this branch, all pass.
- Also run: `scoped-deadline-work.test.ts`,
`delivery-work-coordinator.test.ts`, `deployment-notifications.test.ts`,
`run-start-handover.test.ts`, `heartbeat-run-start-handover.test.ts`,
`heartbeat-run-event-atomic-append.test.ts` (111 tests pass),
`work-signal-debug.test.ts` (12 pass), and `tsc --noEmit` for
`packages/db`.
## Risks
- Mixed deploys, new sender to old receiver: every payload is version 1
and at most 1,024 bytes. The old receiver ignores the `hints` field and
wakes unscoped, as today. A commit with many hints sends several
payloads, so an old receiver can get up to about 7 unscoped wakes for
one commit. The coordinator merges them into one `*` reload.
- The sender bounds bytes, and old receivers compare `.length` in UTF-16
code units. A string never has more UTF-16 code units than UTF-8 bytes,
so a payload of at most 1,024 bytes always passes the old check.
- PostgreSQL collapses identical NOTIFY payloads in one transaction. The
chunks of one topic always carry different hints, so PostgreSQL drops
none of them.
- Mixed deploys, old sender to new receiver: the old payload has no
`hints`, so the new receiver wakes unscoped, as today.
- I did not use one large payload of about 7 KB: current receivers drop
any payload over 1,024 characters before they parse it, so they would
miss the wake during a deploy overlap.
- Consumers that read hints (for example `chat-action-work`,
`approval-queue-seeding-work`) now get the same hints from another
process that they already get from a local commit. Before, they got no
hints from another process.
- The receiver accepts hint strings from the database channel. Consumers
decode them in the same way as local hints. `scoped-deadline-work`
already turns a decode failure into a `*` retry.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edit, test execution).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
819f61f1fd |
fix(evals): repair four cohort-15 chat cases and prefix board links in completion feedback (#21682)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The chat-feature eval catalog runs real chat journeys. It grades saved state, rendered links and run ledgers. > - Chat-eval cohort 15 (OpenCode with `openrouter/deepseek/deepseek-v4.1-flash-1017`, master `36101ebcb9`) failed four cases where the agent did the right thing or where our own server text misled it. > - One failure was a harness race introduced by #21667. Two failures came from graders that read free-form text with keyword regexes. One failure came from an unprefixed approval link in the server's completion feedback, plus a grader that required a literal prefix. > - This pull request fixes the harness step, replaces the free-form graders with saved-state and clicked-link checks, and makes the server emit company-prefixed board links. > - The benefit is that these cases measure the feature again. The retained correct outcomes pass, and the relevant broken outcomes still fail. ## Linked Issues or Issue Description Refs #21667 **What happened?** In chat-eval cohort 15, four cases failed: - `projects.share-project`: the harness followed the agent's ID-based project link. The app then rewrote the URL to the project slug and redrew the page. The owner step timed out because Riley's option in the people picker was "detached from the DOM". #21667 introduced this path. - `skills.author-skill`: the saved skill was a correct, richer checklist. Per-sentence keyword regexes rejected it. For example, "ignore the mouse" tripped the forbidden word "ignore". - `inbox-search.show-decisions`: the reply linked the right approval and tasks and said the question was "addressed to a different teammate". A regex required the word "Riley". - `agents.hire-needs-approval`: both replies linked `/approvals/<id>` without the company prefix. The agent copied that link from `server/src/services/native-runtime/native-completion-feedback.ts`. The grader required a literal `/<prefix>/approvals/<id>`, although the board redirects the unprefixed route. **Expected behavior** Correct agent behaviour passes. Grading uses saved state, setup literals and rendered or clicked links, not free-form wording. The server gives agents canonical company-prefixed board links. **Steps to reproduce** 1. Use the retained cohort-15 evidence for the four cases. 2. Run the old graders on the retained skill markdown and decisions reply. Both fail. 3. For share-project, open a project configuration page by ID while the app rewrites the route after a delay, then use the access picker before the rewrite completes. 4. For hire-needs-approval, finish a task with a pending linked approval. The feedback links `/approvals/<id>`. **Paperclip version or commit** master `36101ebcb9` (cohort 15); fixed on top of current master. ## What Changed - `projects.share-project` (`internal-privacy.ts`, `runtime-support.ts`): after the harness follows the agent's link, it waits until the URL settles on the project's canonical route key. This reuses the #21667 approach for agent profiles: `settleAgentProfileRoute` now calls a shared `settleRouteKey`, and `projectRouteKey` follows the app's `projectRouteRef`. The owner's picker selection (`addProjectAccessMember`) now retries as one unit. If the dialog or picker closes, or an option detaches, it reopens them, searches again and selects again. It continues only when the picker shows the person and Add is enabled. - `skills.author-skill` (`integrator-skills.ts`): removed the keyword-regex checklist grader, the waiver regexes, and the reply regexes for "global activation" and "attached to me" claims. The saved-state checks stay: one agent-created writable skill, its current revision authored by the responding agent, the same bytes on the skill and the revision, the Skills/Studio inspection with reload, and assignment scope from saved assignments and audit rows. The checklist check now requires the setup literals `375`, `WCAG` and `AA` as whole tokens in the saved SKILL.md and in the rendered editor. Whether the skill still asks for keyboard-only navigation and visible labels, and whether it waives a check, is left to human review. Oracle version `skill-authoring.v10`. - `inbox-search.show-decisions` (`internal-decision-search.ts`): removed all reply regexes, including the `Riley` requirement and the wording-based separation checks. The grade now uses saved state (pending decisions unchanged, answered history unchanged, each person's own attention queue and resolver) and rendered links to the exact approval, the question's task and the history task. Oracle version `decision-search.v3`. - Product (`native-completion-feedback.ts`): the completion feedback now links `/<prefix>/approvals/<id>` and `/<prefix>/issues/<identifier>`. This fixes the approval link and the four sibling task links in the same file. - `agents.hire-needs-approval` (`runtime-hiring.ts`): the harness now clicks each reply's rendered link to its approval and records where the UI lands. The grade accepts a prefixed link, or an unprefixed link that the board redirect lands on `/<prefix>/approvals/<id>` with the proposed hire visible. Without such a link, the harness still opens the approval directly so later phases keep their evidence, and the grade fails. Oracle version `runtime-hiring.v4`. - No eval YAML changed, so the TestOps manifest does not change. `chat-catalog-descriptors.json` records the three oracle version bumps. ## Verification - Red/green tests (each new test fails on the old code and passes on the new code): - `internal-privacy-share-route.test.ts` (real Chromium): the project page rewrites its route after 0, 400 and 1500 ms, and one case closes the picker while its first results render. With the old flow, the 400 ms rewrite and the closed picker both time out on the option click (the cohort-15 failure). With the fix, Riley is added on the settled route in all cases. - `integrator-skills.test.ts`: the retained cohort-15 SKILL.md passes. The old grader rejects it. A stale skill body, another author, and a lost 375 width still fail. - `internal-decision-search.test.ts`: the retained cohort-15 reply and a reworded reply pass. The old grader rejects the retained reply. A missing or wrong approval, question-task or history-task link fails. - `native-completion-feedback.test.ts`: a pending linked approval produces `[the pending approval](/<prefix>/approvals/<id>)`, and a pending review stage produces a prefixed task link. The old code produces `/approvals/<id>`. - `runtime-hiring.test.ts`: unprefixed links that the redirect lands on the right approval pass. The old grader rejects them. A link that did not land, landed on another approval, was not clicked, or is another href still fails. - `npx vitest run --config tests/runner-e2e/vitest.config.ts chat-catalog`: all passed (one negative control in `internal.test.ts` changed from a reply wording to "approved while listing"). - `pnpm test:e2e:runner:typecheck` and `server` `tsc --noEmit`: passed. - `server` vitest for `native-completion-feedback`, `native-current-wake-completion-feedback` and `heartbeat/run-completion`: 201 tests passed. - Narrow paid run of the four full case ids on this branch: run 43657390762 (results are added in a comment). ## Risks - Low risk. The product change only adds the company prefix to links in agent feedback text. It adds one indexed company read, and only on the branches that return a link. - Less is graded automatically, by design. `skills.author-skill` no longer checks that the skill keeps the keyboard-only and visible-label checks or that it does not waive a check. `inbox-search.show-decisions` no longer checks how the reply names resolvers or separates the person's decision from others' and from history. `skills.author-skill` no longer checks whether the reply describes the assignment scope truthfully. A person reviews this meaning. Saved state and links still catch the plausible broken outcomes listed above. - `agents.hire-needs-approval` now needs a reply link that really opens the approval. A reply without a link to its approval fails as before. ## Model Used - Claude, model ID `claude-opus-5-5` (Claude Code agent with tool use and code execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
ca23330c69 |
fix(recovery): continue a task on its own when its saved files are permanently lost (#21541)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a run stops early, Paperclip recovers its saved files and then continues the task automatically > - Sometimes the sandbox is gone. Then saved-work recovery ends with `source_unavailable` (use another strategy) > - In that state the task showed Retry. Retry worked, because it continues in a fresh workspace. But nothing continued the task automatically > - A task must never need a person's Retry for a system failure. The target is 0% > - This pull request lets the system take the same fresh-workspace step that Retry takes, when the saved files are permanently lost > - The benefit is that these tasks continue on their own, and the sweep reports them correctly ## Linked Issues or Issue Description Refs #21478 **What happened?** A legacy run on a remote sandbox failed in setup. Its unsaved files were held for saved-work recovery. The sandbox was gone, so the recovery ended `source_unavailable` / `use_alternative_strategy`. The task then showed Retry. For about 2 hours, the assigned-work sweep logged `retry_offered` with outcome `not_started` every 15 minutes. A person had to press Retry. **Root cause** The fresh-continuation fallback returned `person_owned` / `retry_needed`. It does this when a later run of the task was withheld at dispatch (here, a scheduled retry that the hold blocked) and the source run does not continue automatically. The source was a person's earlier Retry that failed with `setup_failed`, so it was not an automatic recovery run and not a restart-class failure. Without the withheld run, the same task already continued automatically. Restore-hold ownership (`isOwnedReconciliationHold`) and the legacy refusal in `inspectFallback` were not the blocker. **Expected behavior** When saved-work recovery proves that the files are permanently gone, the system continues the task in a fresh workspace on its own. This is the same path that Retry takes. ## What Changed - `fresh-continuation-fallback.ts`: the eligibility rule (`interruptionEligibility`, which the fallback and the backlog pass share) also accepts a system-owned source run when a hold of that run has saved-work recovery `exhausted` with a permanent unverifiable-source failure (`UNVERIFIABLE_SOURCE_CODES` × `PERMANENT_SOURCE_RETRY_CLASSES`). - `recovery/assigned-work-sweep.ts`: admission can settle the hold and coalesce the sweep's wake into a pending fresh continuation. The sweep now reports this as `repaired` / `queued` (`continuedFromSavedWork`). Before, it reported `enqueue_refused` / `not_started`. - `legacy-execution-recovery.ts`: the stranded sweep calls `terminalizeLegacyExecution` for the same failed run on every pass. Its restore-hold branch rewrote the hold every time, with a new `automaticRecovery.recordedAt`, a new `updatedAt`, and `continuationDelivery: "invalidated"`. The branch now returns early when the hold already records the same no-replay decision for this run, or when a fresh continuation took the hold over. Finding: a continuation that was already recorded was not clobbered, because the existing `executionReconciliation.runId` guard returned early. But a hold with a `coalesced` delivery state would have been reset, and the churn kept bumping `updatedAt`. - New regression test `legacy-source-unavailable-continuation.test.ts`. The existing `assigned-work-invariant` test now expects the corrected sweep outcome. ## Verification - Red/green: `legacy-source-unavailable-continuation.test.ts` uses `failedRemotePreparation()` with a `destroyed` termination receipt, `recoverRetainedWorkspace`, a withheld scheduled retry, and a source that was a person's Retry. Then it runs `assignedWorkSweep(...).checkIssue` with the real `heartbeatService(db).wakeup`. - Without the fix: outcome `enqueue_refused`, and the fallback returns `person_owned` / `retry_needed`. - With the fix: outcome `repaired`. A stranded pass keeps `continuationDelivery: "pending"`. `deliverReconciledExecutions` starts the successor run. The task no longer shows Retry. There is no `not_started` entry. - Red/green: the "revisits" test fails without the `terminalizeLegacyExecution` guard, because the hold evidence changes. It passes with the guard. - Locally passing: `fresh-continuation-fallback`, `fresh-continuation-move-forward-offer`, `fresh-continuation-retry-delivery`, `legacy-execution-recovery`, both `assigned-work-invariant` suites, `assigned-work-stalled-queued-run`, `assigned-work-sweep-round-trips`, `saved-work-recovery-overdue`, and the new test file (10 files, 225 tests). `tsc --noEmit` passes for `server`. - Not run locally: the full suite. CI covers it. ## Risks - Behavior shift: a task whose source failure is system-owned now continues automatically after its saved files are proven permanently gone, even when an earlier wake was withheld. The new run does not replay the old run. It starts in a fresh workspace, with the same notes and quarantine that Retry uses. Person-owned failure codes still wait for a person. - The new hold guard can skip a rewrite. It skips only when the hold is already resolved with the same no-replay decision and the same restore-recovery data. A changed decision is still written; the test covers this. - No schema changes. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, local Vitest against embedded Postgres). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
68f641ba0b |
fix(recovery): continue a native run cut off by a sandbox client timeout (#21736)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The native runtime runs an agent's provider turn in a sandbox (for example Daytona) and records each failure on the run and its native coordinator > - A Daytona SDK request can hit its client-side deadline during a provider turn. The run then fails with `native_session_interrupted` in an ambiguous state, because the turn started and its outcome is unknown > - That failure blocks the task under a board-owned watchdog hold. No recovery owner selected the hold, because its cause is not an execution-reconciliation cause. The task never ran again, and only a person's Retry could continue it > - Recovery from an infrastructure fault must work automatically, without a person > - This pull request lets the execution-disposition pass hand that hold to the existing fresh-continuation fallback > - The benefit is that the task continues once, in an isolated fresh run, and the cut-off turn is never replayed ## Linked Issues or Issue Description **What happened?** A native run on a Daytona execution environment (native runtime, an OpenRouter model) started its provider turn. A Daytona request then failed with `HTTP request timed out after 300000ms waiting for a response. This is a client-side deadline (DaytonaConfig.requestTimeoutMs, or the per-call operation/execution timeout); any operation already started on the server may still be running.` The run failed with `native_session_interrupted`. The task had no further run: no automatic retry and no continuation. **Expected behavior** The task continues automatically, exactly once, in a fresh run from its saved work. The cut-off provider turn is not replayed. **Steps to reproduce** 1. Start a native run for an assigned task. Let its provider turn start (a `turn.started` event is saved). 2. Make a sandbox request in that turn fail with the Daytona client-side timeout error. 3. The executor saves the coordinator as `terminal_failure` with `recoveryMode: ambiguous_state`, blocks the task, and writes an active board hold with cause `native_session_interrupted`. 4. Run the execution-disposition worker (`settleUnrecoverableExecutions`). Before this change, nothing happens and the task stays blocked. **Paperclip version or commit** `master` at |
||
|
|
01c1e708e9 |
fix(runner): stop a turn that keeps resending a refused tool call and continue fresh (#21734)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native agent runs call Paperclip tools, and
`RepeatedToolFailureGuard` refuses an exact repeat of a call that
already failed validation twice
> - The refusals are not counted and never end the turn, so a model
whose host drops an argument can resend the same refused call until the
run's wall clock
> - In a recorded restart-chaos wave, an opencode/DeepSeek agent sent
the same `PATCH /api/issues/{id}` through `call_api` 431 times in 40
minutes without `body`; 427 were refused. The run held an agent slot and
delayed other tasks
> - The system knows the next step (stop the turn and continue on a
fresh session), so the harness must own it, with no Retry banner
> - This pull request counts refusals, stops the provider turn after a
bounded number with the classified cause `agent_tool_call_loop`, and
queues an automatic fresh-session continuation that tells the model to
use a dedicated tool
> - It also records the declared argument names that each opencode tool
call delivered, so we can tell whether the bridge or the model drops
`body`
> - The benefit is that a looping agent frees its slot in seconds
instead of 40 minutes, and the task continues by itself
## Linked Issues or Issue Description
**What happened?**
An opencode agent on DeepSeek created two subtasks and then tried to set
their descriptions with `call_api` `PATCH /api/issues/{id}`. Every call
arrived with `operationId`, `pathParams`, `query` and `responseText`,
but without `body`. The guard refused each repeat. The model said "I
need to include body" more than 20 times and kept sending the same call:
431 calls, 427 refused, over 40 minutes. The run stayed active and its
agent slot delayed other queued work.
**Expected behavior**
After a small, bounded number of refusals, Paperclip stops the provider
turn and continues the task automatically on a fresh session, with a
hint to use a dedicated Paperclip tool. The board shows no Retry banner
because the system owns this recovery.
**Steps to reproduce**
1. Run a native agent whose model resends one tool call with a missing
required argument.
2. The guard refuses the third and later identical calls.
3. On master, the refusals never end the turn.
**Paperclip version or commit**
`master` at
|
||
|
|
c7efca1eaf |
perf(recovery): read an agent's own reporting chain and each pass's review waits once (#21724)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The server runs its background recovery queues on one Node main thread. Each database statement and each byte returned costs main-thread time. The target is 1,000 concurrent agents per server. > - On the idle hosted staging instance, the stranded-task check is the top remaining cost. Each check read the full agent row (with its configuration blobs) and then every agent of the company, only to decide whether this one agent can run. Per check it also opened a repeatable-read budget transaction for the same company and agent. > - The org-chain evaluation only looks at the agent's own managers. Reading the whole company is work proportional to the size of the company, for an answer that depends on a few rows. > - The same check also read the external review wait of each in-review task one task at a time. > - This pull request reads only the agent's reporting chain (one recursive query up `reports_to`), gives the stranded check a lean agent read, and reads the review waits of a pass's tasks in one call. > - The benefit is half the bytes per check (17.2 KB to 8.6 KB), 87 percent fewer rows, and no per-task review-wait chain for tasks that wait, with the same answers. ## Linked Issues or Issue Description No issue exists. This describes the problem with the fields of `.github/ISSUE_TEMPLATE/enhancement.yml`. **What existing behavior does this improve?** The per-task reads of the stranded-task recovery check (`reconcileStrandedAssignedIssues`) and of every caller of `evaluateAgentInvokabilityFromDb` and `agentService.getById`. **Subsystem affected** Server: issue recovery and agent invokability. **Current behavior** `evaluateAgentInvokabilityFromDb` reads `id, company_id, name, reports_to, status` for every agent of the company on each call. `getById` does the same for `orgChainHealth`. The stranded check calls it once per task, after a full-row agent read with the configuration blobs. On staging at idle these are the `agents where company_id = $1` read at 4.7/s and the full `agents where id = $1` read at 4.2/s. The check also calls `listExternalReviewWaits` for one task at a time (the `chat_message_links` / `chat_github_reviews` reads at about 3/s on staging). **Proposed behavior** Read only the agent's managers, with the same cycle, missing-manager, cross-company and terminated rules. Read nothing when the agent has no manager or when its own status already decides the answer. Read the stranded check's agent with only the fields it uses. Read the review waits of all tasks of a pass in one call, and use that answer only to skip a task that waits. **Reason and benefit** At idle, the average result size on staging is about 17 KB per statement. The stranded check is about 22 statements and 17 KB per task, and it runs about 3 times per second there. ## What Changed - `server/src/lib/agent-reporting-chain.ts` (new): `readAgentReportingChain` reads the agent's managers with one recursive CTE up `reports_to`, limited to the agent's company. `UNION` stops the walk at a cycle. A root agent costs no statement. - `server/src/services/agent-invokability.ts`: `evaluateAgentInvokabilityFromDb` evaluates against the chain, not the company list. It reads nothing when the agent's own status is not invokable, because the status then decides the answer and the chain is not part of it. Every caller benefits with no change: wake enqueue, queued-run claim, retries, run dispatch, wake-queue promotion, continuation delivery, task watchdogs, thread interactions, native completion feedback, native session resumption, native review participants, agent conversations and recovery. - `server/src/lib/agent-records.ts`: `getById` computes `orgChainHealth` from the chain, not from the company list. - `server/src/services/recovery/service.ts`, stranded check: - The agent read selects only `id, company_id, name, reports_to, status, lifecycle_state, lifecycle_holds`, not the full row. - `hasPersistedDurableWaitPath` takes the pass. When the task is in the pass, the first in-review task reads the external review waits of every task of the pass with one `listExternalReviewWaits` call. A task in that answer waits and is skipped. A task not in that answer is read again alone, exactly as before, so a missing answer never lets a check go further than a fresh read would. - A company-wide call without a pass gets a pass of its in-review candidates. - `server/src/services/scoped-deadline-work.ts`: an optional `beforeChecks(entries)` hook gets the entries of the batch before their checks run. - `server/src/services/recovery/pass-reads.ts` (new) and `server/src/services/recovery/work.ts`: the task queues build a pass from the batch in `beforeChecks` and give it to each check of that batch. - Tests: - `server/src/__tests__/agent-reporting-chain.test.ts` (new): property test, see Verification. - `server/src/__tests__/issue-recovery-pass-reads.test.ts` (new): statements and bytes per check, per pass and per full worker drain. - `server/src/__tests__/issue-recovery-review-wait-pass.test.ts` (new): review-wait answers and reads, see Verification. - `server/src/__tests__/monthly-spend-service.test.ts`: the select-sequence stub no longer serves a company agent list to `getById`. - `server/src/__tests__/helpers/recovery-population.ts` (new): seeds a company with an org chart, review tasks with succeeded native runs, and optional paused, terminated and recovery-action tasks. - `server/src/__tests__/issue-recovery.test-fixture.ts`: an optional statement meter, installed before the first connection. - `server/src/__tests__/agent-record-read-projections.test.ts`: `getById` now makes one typed `agents` select; the chain is a raw query. ### Why the shared review-wait read cannot start or escalate work that a fresh read would not - A pass is one coordinator batch: at most 25 checks, one after the other. Its read is never older than the start of the batch. Nothing is kept across batches. - Only the positive answer is used: "this task waits on an external review or reply", which leads to a skip. A skip only delays the task to its next check, and the end of a wait sends its own hint. - A negative answer is not used. The task is read again alone, so every decision that goes further (an enqueue or an escalation) uses a fresh read, as today. - A first version also kept the agent's invokability and the budget answer for the pass. Review found that a kept permissive answer can reach an escalation, which does not re-check those facts the way an enqueue does. That reuse is removed: those reads stay per task. No authorization, pause, approval or budget check is removed or reordered. ## Verification - Equality proof (`agent-reporting-chain.test.ts`, embedded PostgreSQL): - 60 generated org charts, 2 companies each, up to 45 agents. Shapes: deep chains (depth 44), trees, full cycles and mixed graphs. - Managers are missing (inserted with foreign keys off), in another company, terminated, paused or pending approval at random depths. Agent statuses include an unknown value. - For each agent, the test compares the old list-based result with the new result. `evaluateAgentInvokabilityFromDb` deep-equals `evaluateAgentInvokability(agent, companyRows)`. `getAgentWorkEligibility` on the chain deep-equals it on the company list (this is the `getById` path). The test also asserts that every block reason and every chain-health reason occurred. - The list-based evaluation has no depth cap (`reporting_chain_too_deep` is declared but never returned), so the chain read has none either. A cycle ends the walk in both. - Mutation check: when the recursive step does not walk up, both tests fail. - Review-wait equality (`issue-recovery-review-wait-pass.test.ts`, embedded PostgreSQL): two tasks with a direct chat reply wait, one task with an inherited GitHub review wait, one task with none, and one task with an inherited GitHub review wait in another company. One call for all of them gives the same per-task answer as one call per task, for both companies. - Review-wait reads per pass (same file): the pass makes 2 bound-run message reads (one shared, one for the task whose candidate is not a wait) where per-task reads make 3, and 89 statements instead of 91. The check results are identical. On master the pass is ignored, so the test fails (3 is not less than 3). - Statements and bytes (`issue-recovery-pass-reads.test.ts`, 50 agents, 200 review tasks with succeeded native runs and no review waits, embedded PostgreSQL, statement meter on the client sockets): | | master | this branch | | --- | --- | --- | | One check: statements | 22 | 22 | | One check: bytes received | 17,170 | 8,617 | | One check: rows | 57 | 8 | | 25 checks: bytes | 429,331 | 215,505 | | Real worker, full company drain (200 checks): statements | 4,414 | 4,422 | | Real worker, full company drain: bytes | 3,455,957 | 1,749,701 | | Real worker, full company drain: rows | 11,600 | 1,800 | | Company agent-list reads per drain | 200 | 0 | Here no task waits, so each batch pays one extra review-wait statement (8 per drain). Where tasks wait, the shared read replaces the per-task review, publication and authorization reads. - Regression after the review change, embedded PostgreSQL (664 tests): scoped-deadline-work, heartbeat-process-recovery, issue-recovery-deadlines, assigned-work-invariant, recovery-deadline-acceptance, heartbeat-issue-liveness-escalation, slack-conversation-lifecycle. Also issue-list-review-wait-authorization, external-review-wait, monthly-spend-service. - Regression of the first version, embedded PostgreSQL (757 tests): issue-assignee-invokability-routes, agent-record-read-projections, issue-graph-liveness-equivalence, heartbeat-paused-assignee-new-task, heartbeat-process-recovery, heartbeat-issue-liveness-escalation, assigned-work-invariant, recovery-deadline-acceptance, issue-recovery-actions, heartbeat-refused-assignment-wake, failed-continuation-continues-again, agent-invokability, agent-cross-tenant-authz-routes, heartbeat-stop-queued-task, issue-tree-pause-resume-run. - `tsc --noEmit -p server/tsconfig.json` passes. ## Risks - A test double that gives `evaluateAgentInvokabilityFromDb` an object with only `select` now needs `execute`, and a select-sequence stub of `getById` gets one select fewer. CI found one (`monthly-spend-service`), fixed here. - A batch with no waiting task pays one extra cheap review-wait statement. - This branch and #21729 both touch `server/src/services/recovery/work.ts` in nearby lines. The second one to merge may need a small rebase. - `server/src/services/recovery/service.ts` was already past 1,000 lines. This change adds about 25 lines to it, beside the check it changes. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, local PostgreSQL test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
41a3520a2f |
fix(issues): count a blocker covered when its wake already created a queued run (#21731)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A blocked parent task shows "covered" when its blockers have work
that will start by itself, and "needs attention" when a person must act
> - The blocker-attention read counts a blocker as active only through
its current execution run, or through a waiting wake that has not
created a run yet
> - A child whose wake already created a queued run that has not started
matches neither, so the parent shows "needs attention" and lands in the
blocked inbox
> - That work starts by itself when an agent slot frees, so the
attention is false and asks a person to act for no reason
> - This pull request also counts a blocker as active when its waiting
wake points at a queued or running run
> - The benefit is that the board and the blocked inbox stop showing
false attention while queued child work waits for a slot
## Linked Issues or Issue Description
**What happened?**
A parent task was blocked by two child tasks. Each child had a queued
run that a wake had created, and the runs waited for agent slots. The
parent showed `needs_attention` / `attention_required` and appeared in
the blocked inbox. The children started about 25 minutes later and the
parent then finished without a person.
**Expected behavior**
The parent shows `covered` with reason `active_child` while a child's
queued run waits to start.
**Steps to reproduce**
1. Create a blocked parent task with one child blocker that is assigned
to an agent.
2. Insert a queued wake for the child whose `runId` points at a queued
`heartbeat_runs` row, and leave the child's `executionRunId` empty.
3. List blocked tasks. The parent shows `needs_attention`.
**Paperclip version or commit**
`master` at
|
||
|
|
61c315ebab |
perf(chat): keep GitHub review check deadlines on company-wide rebuilds (#21725)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The GitHub chat channel shows a check run on each pull request it reviews. The review-check sweep (`review-check-work.ts`) keeps one `github_review_check` action per check. It republishes the check when its status changes (queued, running, completed) and polls the pull request head on a saved deadline. > - Each time the sweep loads its checks, it computes the current check status of every `received` check (`githubReviewCheckState`: 3–5 statements) to see whether the status changed. > - Run lifecycle changes anywhere in a company send a company-wide hint. So the sweep rebuilds every company's checks every 30 s and recomputes the status of every waiting check, even when nothing about those checks changed. > - This pull request keeps the saved poll deadline on a company-wide rebuild and reads the status only on targeted hints, at startup, and in `check()`. > - The benefit is that an idle company with many waiting checks costs one page read per rebuild instead of 3–5 statements per check. ## Linked Issues or Issue Description Refs #21705, #21711 (the same class of idle reload in the GitHub response sweeps). No issue exists. This describes the problem with the fields of `.github/ISSUE_TEMPLATE/bug_report.yml`. **What happened?** On paperclip.staging at |
||
|
|
5acf27bc3b |
perf(server): read the instance role and membership once per GET request in authorization (#21716)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Every board request goes through authorization, and the server runs
all requests on one Node main thread
> - The cost on that thread is per database statement
> - One `GET /api/issues/:id` makes several authorization decisions (the
issue, its project, mentioned projects, related task edges), and each
decision reads `instance_user_roles` and `company_memberships` again
> - On a task with a blocker and a project mention, that is 4 role reads
and 4 membership reads in the authorization service for one request, all
with the same answer
> - This pull request shares those two reads among the decisions of one
GET or HEAD request
> - The benefit is fewer statements on every polled board read, with no
authorization cache across requests
## Linked Issues or Issue Description
**What existing behavior does this improve?**
`authorizationService.isInstanceAdmin` and `getActiveMembership` read
the database for each decision in a request.
**Subsystem affected**
Server: authorization service and app middleware.
**Current behavior**
`GET /api/issues/:id` for a plain task sends 3 `instance_user_roles`
reads and 3 `company_memberships` reads (1 each in the session
middleware, 2 each in authorization). A richer task sends 5 and 5.
**Proposed behavior**
A new middleware, `authorizationReadScope`, opens a per-request memo for
GET and HEAD requests. For a session actor, it seeds the instance admin
answer that the session middleware read at the start of the same
request. Inside the scope, `isInstanceAdmin`, `getActiveMembership`,
`issueReadSqlCondition` and `projectReadSqlCondition` read each key
once. The plain task now sends 1 role read (the session middleware) and
2 membership reads (1 in the session middleware, 1 in authorization).
The issue list and summaries routes also drop their extra role reads.
Other methods, background work and other callers keep the old reads. The
memo ends with the request, so a revoked membership takes effect on the
very next request.
**Reason and benefit**
Fewer statements per polled read, with the same decisions.
**Breaking changes**
None.
## Memo lifetime rule
- The memo exists only for GET and HEAD requests.
- It wraps `res.writeHead`, `res.flushHeaders` and `res.write` once and
never reassigns them, so hooks that later middleware installs on the
response stay in place.
- It closes at the first of: `res.writeHead`, `res.flushHeaders`,
`res.write`, `finish` or `close`. After it closes, every read goes to
the database. A stream, a long poll, a download or a callback that runs
after the response therefore never uses the memo.
- Each entry is used for at most 2 seconds
(`AUTHORIZATION_READ_MAX_AGE_MS`), then read again. Correctness does not
depend on a list of routes.
- A rejected read is removed from the memo, so the next decision reads
again.
- Nothing is kept across requests.
Long-lived and streaming routes that I checked:
- `GET /plugins/:pluginId/bridge/stream/:channel` (SSE): authorizes once
at connect with `assertBoardOrgAccess` and `assertCompanyAccess` on
`req.actor`. It does not re-authorize per event. The memo closes at
`writeHead`.
- Streaming downloads (`object.stream.pipe(res)` in assets, access and
issue attachments): they authorize before the stream starts. The memo
closes at the first write.
- `POST /board/chat/stream` and `POST
/companies/:companyId/skill-sources/discover` (NDJSON): POST, so no
memo.
- Live events, plugin API, runner and terminal WebSockets: they use
`server.on("upgrade")`, so they do not pass through Express middleware.
## What Changed
- New `server/src/services/authorization-read-scope.ts`:
`authorizationReadScope` (an AsyncLocalStorage scope for GET and HEAD)
and `readOncePerRequest`.
- `authorizationService.isInstanceAdmin` and `getActiveMembership` use
`readOncePerRequest`. The existing batch snapshot memo still comes
first.
- `issueReadSqlCondition` and `projectReadSqlCondition` use the same
`admin:<userId>` key for their instance admin check.
- For `source: "session"` board actors, the scope seeds `admin:<userId>`
with `req.actor.isInstanceAdmin`, which the session middleware read from
`instance_user_roles` in the same request.
- `createApp` mounts `authorizationReadScope` after the actor and Cloud
control middleware.
- `authorizationReadScope` closes the memo when the response starts or
ends, ignores entries older than 2 seconds, and drops rejected reads.
- New exported `readInstanceAdmin(db, userId)`: the one predicate for
`isInstanceAdmin`, `issueReadSqlCondition` and
`projectReadSqlCondition`.
- New test `authorization-request-reads.test.ts`: one role read and one
membership read in authorization per issue read, and a removed
membership gives 404 on the next request.
## Verification
- Red: without the scope, the issue read sends 3 `instance_user_roles`
statements (`expected ... to have a length of 2 but got 3`).
- Green: with the scope, it sends 1 role read and 2 membership reads.
The issue list plus the summaries request send 1 role read each.
- Revocation: delete the membership row, and the next GET returns 404.
HEAD returns 200, then 404 after the revoke.
- Stream: a GET that flushes headers, then re-reads the membership after
it is deleted mid-stream, sees the deletion.
- Background callback: a callback scheduled in a GET that runs after the
response and after a revoke reads again and sees the deletion.
- Age bound: an entry older than 2 seconds is read again. A rejected
read is not kept.
- Seed predicate: for users with an `instance_admin` row, another role
row, and no row, the session middleware value equals
`readInstanceAdmin`.
- Other actors: an agent key, a board key and a cloud tenant actor get
no seeded value.
- Response hooks: a later middleware that wraps `res.writeHead` and
`res.write` keeps working for JSON and streamed responses. This test
fails with a version that restores the original methods.
- Red: with the previous implementation, the stream, callback, age and
rejection tests fail.
- `pnpm exec vitest run` on `access-routes-hidden-floor`,
`access-routes-permissions-upgrade`, `access-service`,
`agent-auth-middleware`, `authorization-privacy-query`,
`authorization-request-reads`, `authorization-run-context`,
`authorization-service`: 8 files, 177 tests pass. Also
`inbox-mine-candidates`, `issue-read-filter-batching`,
`attention-summary-counts`, `issues-unread-inbox`,
`live-runs-batched-reads`, `privacy-production-review`,
`activity-routes`, `authorization-privacy-query`,
`muse-native-visibility`: 224 tests pass (one
`privacy-production-review` test failed once under machine load and
passed on two reruns).
- `pnpm --filter @paperclipai/server typecheck` passes.
## Risks
- This changes where authorization gets two facts inside one request, so
it needs a human review before merge.
- A GET route that changes a membership or an instance role and then
authorizes again before its response starts, within 2 seconds, would see
the earlier value. GET routes do not change memberships or roles. POST,
PATCH, PUT and DELETE requests do not use the memo.
- A board actor built by the session middleware still carries its own
connect-time `memberships` and `isInstanceAdmin` for the life of a
stream. That is the existing behavior, and this change does not affect
it.
- Nothing is cached across requests.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, test runs).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
a78be8b601 |
feat(native): send launch preparation and run digests to plugin task prepare (#21717)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Environment plugins can run a native Runner as a plugin task, and a warm session keeps that Runner between turns > - On each warm turn Core sends the plugin a `prepare` operation with the full launch requirements for the new run > - The requirements' `requirementsDigest` covers the run identity, and the run identity changes on every turn > - So a plugin cannot tell that only the run moved, and it must redo its whole preparation every warm turn > - This pull request gives the plugin two digests: one that stays the same while its preparation stays valid, and one that names the run > - The benefit is that a plugin can transfer only the run identity when nothing it reconciles changed ## Linked Issues or Issue Description **What existing behavior does this improve?** The `prepare` operation that Core sends to an environment plugin for a plugin Runner task, on cold starts and on warm attaches. **Subsystem affected** Native runtime, plugin Runner tasks, the Runner launch contract, and the plugin SDK environment task contract. **Current behavior** A warm attach sends the full requirements. The only digest available, `requirementsDigest`, includes `runId`, `turnId` and `itemId`, so it changes on every turn even when the launch is otherwise the same. **Proposed behavior** Core computes two digests for every `prepare`: - `preparationDigest`: the launch requirements an environment reconciles when it prepares a Runner: - the runner, lease and session identity; - provider, model and settings; - the prompt and instruction text; - the runtime context's instruction and skill bundles, working copy and connection instructions. It leaves out the `runId`, `turnId` and `itemId`, and the runtime context's MCP assignment and binding (with `aggregateDigest`, which hashes the assignment). MCP reaches the Runner through Core's tool authority on every run attachment, never through files the environment publishes. Core already continues a warm session across an MCP-only change (`nativeRuntimeContextsMatch` with tool refresh). The binding is also named per run (`native-mcp:<runId>`). So the digest stays the same across warm turns of one Runner, lease and session, including when the agent's MCP access changes. - `runDigest`: `preparationDigest` plus everything it leaves out: `runId`, `turnId`, `itemId`, and the MCP assignment and binding. It is unique to one run's launch. A plugin that advertises `launchDigestVersion: 1` in its task description gets both fields on `prepare`. **Reason and benefit** A plugin can compare `preparationDigest` with its saved preparation and transfer only the run identity when they match, instead of a full preparation. **Breaking changes** None. Plugins without `launchDigestVersion` get no new fields. ## What Changed - `packages/paperclip-runner/src/live/runner-provider-launch.ts`: add `runnerLaunchPreparationDigest` and `runnerLaunchRunDigest`. The preparation digest drops the MCP assignment and binding; the run digest takes the full requirements and includes them. Both use the same canonical JSON as `runnerLaunchRequirementsDigest`, with their own schema tags so the three digests never collide. Export them from the package and the server's vendored runner shim. - `packages/plugins/sdk/src/environment-tasks.ts`: add optional `preparationDigest` and `runDigest` to the `prepare` operation, and optional `launchDigestVersion: 1` to the task description. The capability gates both digests. - `server/src/services/native-runtime/plugin-runner-task.ts` and `native-session-executor.ts`: send both digests on `prepare` when the description advertises `launchDigestVersion: 1`. - `server/src/services/environment-task-runtime.ts`: recompute both digests from the requirements, and refuse a `prepare` whose digests are missing in part or do not match. - `tests/runner-e2e/source-sdk-public-inputs.json`: refresh the SDK public source pin. - Tests: - the preparation digest is stable across runs and changes when the runner, lease, session, instructions, model, approval policy or settings change; - with a runtime context, an MCP-only change (the run's binding, no binding, a changed access set) keeps the preparation digest and changes the run digest; - a change to the instruction bundle, instruction entry, a skill bundle, the skill set, the working copy or the connection instructions changes the preparation digest; - the run digest changes per run; - Core sends the digests only when advertised; - the runtime refuses forged or partial digests before it calls the plugin. ## Verification - `npx vitest run src/live/runner-launch-preparation.test.ts` in `packages/paperclip-runner`: 26 passed. - `npx vitest run src/services/native-runtime/plugin-runner-task.test.ts src/__tests__/environment-task-runtime.test.ts` in `server`: 160 passed. - `npx vitest run source-sdk-public-inputs.test.mjs source-sdk-evidence.test.ts source-sdk-public-type.test.ts` in `tests/runner-e2e`: 59 passed. - `pnpm typecheck` in `server` and in `packages/paperclip-runner`: clean. - Not run locally: the full suite. CI covers it. ## Risks Low risk. The digests are new optional fields behind a plugin capability version. Core computes them and checks them again before it calls the plugin, so a plugin can rely on them. They prove only that the launch requirements are unchanged. A plugin that skips work on a match must still check its own state, such as files it published into the sandbox. If the runtime context carries values that change per run, the preparation digest changes too, so a plugin does its full preparation. That is safe, but the plugin saves nothing. The MCP binding is the one such value Core sets per run today, and MCP is excluded entirely. A plugin that does publish MCP configuration into its sandbox must not rely on this digest for it. Values outside the requirements are not covered at all: the account identity and credentials, the GitHub broker, project IDs, and the plugin's own runtime revision. A plugin must keep checking those itself. A match lets a plugin skip publishing its launch configuration, not computing the descriptor: `prepare` must still return a launch whose `requirementsDigest` matches the current requirements, which carry the current MCP assignment and binding. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, code search). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
89de6cbe3e |
perf(server): list skills without a folder transaction when bundled folders exist (#21715)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The board UI polls `GET /api/companies/:companyId/skills`, and the server runs all requests on one Node main thread > - The cost on that thread is per database statement, so a polled read must stay small > - Each skills list read reconciles the bundled skill folders, and that opened two transactions with a company advisory lock, even when all folders already existed > - A write-path guard must not run a lock and a transaction on every poll > - This pull request checks the current folders with a plain read first and takes the lock only when a folder must change > - The benefit is a skills list read with no transaction and no advisory lock, and 13 fewer statements (58 to 45) ## Linked Issues or Issue Description **What existing behavior does this improve?** `folderService.ensureBundledCategory` and `pruneEmptyBundledCategories` always open a transaction and take `pg_advisory_xact_lock` for the company folders. **Subsystem affected** Server: skill folders service, used by the company skills list. **Current behavior** Every `GET /api/companies/:companyId/skills` sends 58 statements, including 2 transactions (BEGIN, lock, reads, COMMIT), even when nothing changes. **Proposed behavior** `ensureBundledCategory` reads the company skill folders once. When the bundled root exists and the category folder exists with the expected name, it returns that folder. `pruneEmptyBundledCategories` computes the empty categories with plain reads and returns when there is nothing to delete. In all other cases, both functions take the lock and run the old code, which reads the folders again under the lock. The same request sends 45 statements with no transaction. **Reason and benefit** The polled read stops taking a company lock, and it sends fewer statements. **Breaking changes** None. ## What Changed - `ensureBundledCategory`: a plain read fast path when the category folder and the bundled root exist and the name matches. Create and rename still run under the company folder lock. - `pruneEmptyBundledCategories`: a plain read pre-check. The delete still runs under the lock, and it computes the empty set again under the lock. - New test `company-skills-list-statements.test.ts`: a warm skills list read has no transaction and no advisory lock and at most 45 statements; two concurrent first ensures create exactly one folder; two concurrent first list requests create no duplicate folder; a renamed category is repaired and an empty category is pruned through the locked path. ## Verification - Red: before the change, the warm read had 2 transactions (`expected 2 to be +0`) and 58 statements. - Green: after the change, the warm read has 0 transactions, no `pg_advisory` statement, and 45 statements. The response body is equal to the first read. - Concurrency: two `ensureBundledCategory` calls at once return the same folder, and there is exactly one row for the system key. - `pnpm exec vitest run` on `company-skills-list-statements`, `company-skills-service`, `company-skills-routes`, `company-skills-runtime-concurrency`, `company-skills-catalog-service`, `company-skills` and the folder tests: 215 tests pass. Two `company-skills.test.ts` tests in "project workspace skill discovery" fail on my machine with and without this change. They test file system discovery and do not use the folders service. - `pnpm --filter @paperclipai/server typecheck` passes. ## Risks - Low risk. The fast path returns only when the locked path would return the same folder without a write. Every write still happens under the lock, after a fresh read. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
30c4d2f4a1 |
perf(server): read the issue detail with fewer database statements (#21713)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The board UI loads the task page through `GET /api/issues/:id`, and
the server runs all requests on one Node main thread
> - A CPU profile of the server shows that the cost is per database
statement: SQL building, parameter encoding, result parsing and one
socket write each
> - The issue detail read sent 73 statements for a task with two
ancestors, one blocker and a project mention
> - Several of those statements read the issue row again, only to get
values that the route already loaded
> - This pull request passes the loaded issue to those reads, walks the
ancestor chain in one query, and skips reads whose result the response
does not use
> - The benefit is 9 fewer statements on each task page load (73 to 64),
plus one fewer statement for each extra ancestor level, with a
byte-identical response
## Linked Issues or Issue Description
**What existing behavior does this improve?**
`GET /api/issues/:id` reads the same issue row several times in one
request, and walks the ancestor chain with one statement for each level.
**Subsystem affected**
Server: issue routes and the issue, issue reference and project
services.
**Current behavior**
The route loads the issue, then `getRelationSummaries`,
`getCurrentScheduledRetry`, `findMentionedProjectIds`,
`listIssueReferenceSummary` and `getAncestors` each read the issue row
again. `getAncestors` sends one statement for each ancestor. The inbox
archive fields read two activity aggregates even when the issue has no
archive row. The project embed reads the workspace provider catalog, but
the compact embed does not return it. The test fixture request sends 73
statements.
**Proposed behavior**
The route passes the values it already has. The ancestor walk is one
recursive query. The activity aggregates run only when an archive row
exists. The project embed skips the provider catalog. The same fixture
request sends 64 statements, and the JSON response is the same.
**Reason and benefit**
Fewer statements per request means less CPU on the main thread for every
open task page.
**Breaking changes**
None. Other callers of the changed service functions keep the old
behavior, because the new arguments are optional.
## What Changed
- `issueService.getRelationSummaries` and `getCurrentScheduledRetry`
accept an optional known `companyId` and then skip the issue read.
- `issueService.findMentionedProjectIds` accepts the loaded issue
(`companyId`, `title`, `description`).
- `issueService.getAncestors` accepts the known `parentId` and walks the
chain in one recursive query. It keeps the order (nearest first), the
limit of 50, and the stop at a cycle or at a task that is its own
parent.
- `issueService.getActiveInboxArchiveFields` reads the archive row first
and reads the activity aggregates only when an archive row exists.
- `issueReferenceService.listIssueReferenceSummary` accepts an optional
known `companyId`.
- `issueThreadInteractionService.listForIssue` accepts the loaded issue
(`status`, `companyId`, `executionWorkspaceId`). `GET
/api/issues/:id/interactions`, which the board polls, passes it (11 to
10 statements).
- `projectService.getById` accepts `{ workspaceProvider: false }`. The
issue detail route uses it, because `compactIssueProject` never returns
`workspaceProvider`.
- Test helpers: `createStatementCensus` (statements per request, grouped
by statement shape with the server callers), and a board route fixture
with the real session actor middleware. `meterDatabaseStatements` can
record a call stack for each statement.
- New tests: `issue-detail-statements.test.ts` (statement budget,
byte-identical normalized JSON against a fixture file, access rules,
membership removal on the next request, ancestor walk with a cycle) plus
an interactions poll test, and `board-route-statement-census.test.ts`
(census of the polled and page-load board routes).
## Verification
- Red: before the change, the fixture request sent 73 statements, and
the budget test failed (`expected 73 to be less than or equal to 64`).
- Green: after the change, the request sends 64 statements. The
normalized JSON is byte-identical to the fixture file that I wrote
before the change.
- Access: another company's user gets 404, a missing issue gets 404, an
anonymous request gets 404, a hidden issue stays readable for a member,
and a removed membership gets 404 on the very next request.
- `pnpm exec vitest run` on `issue-detail-statements`,
`board-route-statement-census`, `issues-goal-context-routes`,
`issues-service`, `issue-references-service`, `inbox-archive-routes`,
`heartbeat-task-context`, `project-workspace-provider.integration`,
`workspace-provider-failure`, `issue-telemetry-routes`,
`issue-thread-interaction-routes`, `issue-document-restore-routes`,
`heartbeat-referenced-projects`, `project-environments`: 14 files, 414
tests pass.
- `pnpm --filter @paperclipai/server typecheck` passes.
- Interactions: the polled route sends 10 statements instead of 11, with
a response equal to the service result without the known issue.
- Census: `PAPERCLIP_STATEMENT_CENSUS=/tmp/census.txt pnpm exec vitest
run src/__tests__/board-route-statement-census.test.ts` writes
statements per request and the top statement shapes for each board
route.
## Risks
- Low risk. The new arguments are optional, so other callers keep the
old reads.
- When the route passes a known `companyId`, a concurrent delete of the
issue no longer gives a 404 from these sub-reads. The route already
loaded and authorized the issue in the same request, so the response is
the same in practice.
- The JSON fixture file fails when someone changes the issue detail
response. Regenerate it with `PAPERCLIP_WRITE_ISSUE_DETAIL_GOLDEN=1` and
review the diff.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, test runs).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
f4f4902707 |
perf(chat): read the run state of a GitHub response sweep page in one batch (#21711)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The GitHub chat channel has two sweeps for request response
comments: the acknowledgement sweep (`acknowledgement-work.ts`) and the
terminal-response sweep (`terminal-response-work.ts`). Each reads pages
of up to 100 requests.
> - For each loaded request, both sweeps call `readGitHubResponseRun`
one row at a time. That is one latest-run statement, plus the run
identity context read and the message link read of
`chatRunSourceMessageId`. A page of 100 requests costs about 300
statements.
> - On paperclip.staging, those two provenance reads ran at about 13/s
each. #21705 stops the idle backlog from being reloaded. This PR makes
the reads that remain cost a constant number of statements per page.
> - The per-row read also had a failure mode. A 403 from one run's
provenance check aborted the acknowledgement sweep's whole page, so no
request of that company was acknowledged until the 403 went away.
> - This pull request adds a batched `readGitHubResponseRuns` and uses
it once per page in both sweeps.
> - The benefit is a constant number of statements per page, and a
provenance failure that affects only its own request.
## Linked Issues or Issue Description
Refs #21705 (the same sweeps; that PR stops reloading requests that
cannot progress).
No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/bug_report.yml`.
**What happened?**
On paperclip.staging (2026-11-15, about 01:45–02:25Z),
`provenanceReads.context` (`select … from run_identity_contexts where id
= $1 and run_id = $2 and company_id = $3 and status = $4 limit $5`) ran
at 13.37/s, and `provenanceReads.link` (`select
chat_conversations.issue_id, chat_endpoints.provider from
chat_message_links join … where chat_message_links.company_id = $3 and
comment_id = $4 …`) ran at 13.20/s. Both GitHub response sweeps reach
them through `readGitHubResponseRun` → `chatRunSourceMessageId(db, run,
{ provider: "github" })`, once for each loaded row. `load()` costs one
page read plus three statements per row. `check()` read the run state
twice for each due request (before and after the acknowledgement).
In the acknowledgement sweep, a 403 from `chatRunSourceMessageId` (for
example, a run whose identity context is no longer accepted) threw out
of `entry()` and aborted the whole page. The terminal sweep already kept
such a row due, so the acknowledgement owner could revoke it.
**Expected behavior**
A page of requests costs a constant number of statements. A failure for
one run affects only the requests of that run.
**Steps to reproduce**
1. Create 120 GitHub response requests on 30 tasks. Give each task an
ended run with an accepted identity context.
2. Register the terminal-response sweep. On master, the first pass costs
one page read plus three statements per row.
3. Mark the requests `received`, revoke one run's identity context, and
register the acknowledgement sweep. On master, the load fails with a
403, and none of the 150 requests is acknowledged.
**Paperclip version or commit**
master at
|
||
|
|
8cc26221af |
perf(native): accept a plugin's in-preparation project-auth check on warm attach (#21710)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Environment plugins can run a native Runner as a plugin task, and a warm session keeps that Runner between turns > - On a warm turn Core attaches the kept Runner to the new run: the plugin prepares the new run, then Core checks managed project authentication > - A warm attach has no submit step, so Core sends a separate `check_project_auth` operation after `prepare` > - That separate operation is one more plugin dispatch and Launcher round trip on every warm turn, about 0.12–0.18 s, and it is serial > - This pull request lets the plugin do the check inside `prepare` and say so, so Core does not send the second operation > - The benefit is a faster warm turn with the same proof and the same failure behavior ## Linked Issues or Issue Description **What existing behavior does this improve?** The warm attach of a plugin Runner task (`plugin-runner-task.ts` `prepareLaunch`). When `launched` is true, Core always calls `checkProjectAuth` after `prepare`. **Subsystem affected** Native runtime, plugin Runner tasks, and the plugin SDK environment task contract. **Current behavior** Cold starts fold the check into submit (`projectAuthCheckAtSubmit`). Warm attaches send `prepare`, then a separate `check_project_auth`. Measured on a plugin sandbox, the separate operation costs 0.12–0.18 s on each warm turn. **Proposed behavior** The `prepared` result has a new optional flag, `projectAuthChecked`. A plugin sets it only when it ran the managed project-authentication check inside this preparation, for this launch's configuration, provider and projects, and the check allowed them. Core then records the same lease proof that an allowed `check_project_auth` records, and skips the separate check. **Reason and benefit** One fewer serial round trip on every warm turn. The plugin already runs the check for this task; this removes a dispatch, not a check. **Breaking changes** None. The flag is optional. A plugin that does not set it, or that saw a refusal, gets the current behavior. Plugins detect the flag with the SDK schema before they send it. ## What Changed - `packages/plugins/sdk/src/environment-tasks.ts`: add optional `projectAuthChecked: true` to the `prepared` result, with a doc comment that defines it. - `server/src/services/environment-task-runtime.ts`: when a `prepared` result has `projectAuthChecked`, record `runnerTaskProjectAuthValidation` for the launch's configuration, provider and projects, without `atSubmit`. This is the proof that `submit` and `read_file` require. - `server/src/services/native-runtime/plugin-runner-task.ts`: do not send `check_project_auth` when the prepared result has `projectAuthChecked`. - Tests: a warm attach with the flag makes only `prepare`; without it, `prepare` then `check_project_auth`. The runtime test covers all four combinations of `projectAuthCheckAtSubmit` and `projectAuthChecked`. ## Verification - `npx vitest run src/services/native-runtime/plugin-runner-task.test.ts src/__tests__/environment-task-runtime.test.ts` in `server/`: 157 passed. - Not run locally: full typecheck and full suite. CI covers them. ## Risks Low risk. Core trusts the flag the same way it trusts a plugin's `check_project_auth` answer and `projectAuthCheckAtSubmit` today. A refusal still goes through Core's own check, so the conflict error is unchanged. The plugin side ships separately and sends the flag only when its SDK accepts it. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, code search, tracing queries). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
b5ba548112 |
feat(server): add an instance-admin server metrics endpoint for the scale ramp (#21708)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The scaling plan (`doc/plans/2026-11-13-thousand-concurrent-agents.md`) has a finish line that a real-agent ramp harness checks (#21695) > - Four finish-line criteria need numbers from inside each server process: main-thread CPU, run event lateness, agent API time before the handler, and missed lease renewals > - The server had no way to give them, so every harness run was at best "incomplete" > - This pull request adds one instance-admin endpoint that returns those numbers per process over a short sliding window, off by default, and connects the harness to it > - The benefit is that a ramp run can measure every server criterion it can see, with near-zero cost when on and no cost when off ## Linked Issues or Issue Description Refs #21695 **Subsystem affected** Server (instance routes, run event append, native lease renewal), database client (statement counter), and the scale-ramp harness in `scripts/scale-ramp/`. **Problem or motivation** The scale-ramp harness reads `PAPERCLIP_SCALE_METRICS_URL`, but the server has no endpoint for it. The criteria `main_thread_cpu`, `run_event_lateness_p99`, `agent_api_pre_handler_p99` and `lease_renewals_missed_zero` are therefore always "not measured", and no run can pass. **Proposed solution** Add `GET /api/instance/server-metrics`. It returns a JSON snapshot of one server process over the last 50 to 60 s. Only an instance admin can read it. The env flag `PAPERCLIP_SERVER_METRICS` (read once at start, default off) turns on collection and the route. Off, the path answers 404 and no collector starts. **Alternatives considered** - The postgres.js `debug` option for the statement counter: rejected, because it captures a stack (`new Error()`) for every query. - Process CPU from `process.cpuUsage()` as `mainThreadCpu`: rejected, because it includes GC and libuv threads. It is returned as `processCpu` instead, and `mainThreadCpu` reads the main thread from `/proc` (Linux only, null elsewhere). - Default on: the measured cost is negligible (numbers below), but the event-loop monitor wakes the process every 20 ms when idle, and this is a new instance-wide surface. It stays off; the runbook says how to turn it on for a ramp run. ## What Changed - `server/src/services/server-metrics.ts`: a per-process collector. It uses `performance.eventLoopUtilization()`, `monitorEventLoopDelay` (20 ms), `process.cpuUsage()`, the main thread's `/proc/self/task/<pid>/stat` on Linux, and six 10 s buckets of native `perf_hooks` histograms. One unref'd timer rotates the buckets. Nothing on the hot path awaits, allocates per call or does I/O. - `server/src/routes/instance-server-metrics.ts`: `GET /api/instance/server-metrics`, guarded by the existing `assertInstanceAdmin` (the check that `/instance/database-backups` and other instance routes use). It returns numbers, a random per-process `processId`, `uptimeSec`, `buildRevision` and `windowMs`. It returns no SQL, names, IDs or hostnames. - `server/src/app.ts`: when the flag is on, two middlewares time agent API requests from arrival to the API router, the route is mounted, and shutdown stops the collector. When the flag is off, nothing is mounted. - The snapshot also carries `dedicatedConnections` from `dedicatedDbConnectionStats()` (#21693): `limit`, `inUse`, `waiting` at request time and `peakInUse`, `queued`, `overLimit`, `nestedOverLimit` since start. The harness ignores it today; it shows dedicated-connection queueing in a restart burst. - `packages/db/src/statement-count.ts` and `lazy-transaction-queries.ts`: a Drizzle logger that increments one counter per statement replaces Drizzle's no-op logger. No edit to `client.ts`. - `paperclip-control-plane-port.ts`: one call records runner `emittedAt` to committed row for each committed (not duplicate) runner event. - `native-execution-lease.ts`: one call counts a lease whose renewal deadline passed. - Harness: missed lease renewals are summed per `processId` (several processes, or a restart, no longer cancel out). A metrics URL on the target origin gets the board token in the `Authorization` header; `PAPERCLIP_SCALE_METRICS_TOKEN` still overrides it, and another origin gets no token. - Contract: `scripts/scale-ramp/server-metrics.fixture.json` is the serializer's own output. A server test asserts the serializer produces exactly this file, and a harness test parses it and checks every field. - `rules/company-scope-exceptions.mdc`: adds the snapshot as an instance-wide exception with its reason. The runbook in the plan doc documents the path, the flag, the credential and each field. | Field | Provided | Source | | --- | --- | --- | | `mainThreadCpu` | yes (Linux; null elsewhere) | `/proc/self/task/<pid>/stat` utime+stime | | `eventLoopUtilization` | yes | `performance.eventLoopUtilization()` | | `eventLoopDelayP99Ms` | yes | `monitorEventLoopDelay`, highest 10 s p99 in the window | | `statementsPerSecond` | yes | Drizzle logger counter (not BEGIN/COMMIT or raw driver calls) | | `poolWaitP99Ms` | absent | postgres.js has no cheap hook for queue time | | `runEventLatenessP99Ms` | yes (null with no events) | runner event append | | `agentApiPreHandlerP99Ms` | yes (null with no agent requests) | arrival and router-entry middlewares | | `leaseRenewalsMissed` | yes, counter since process start | native lease deadline timer | ## Verification - `cd server && npx vitest run src/__tests__/instance-server-metrics-routes.test.ts src/__tests__/server-metrics.test.ts`: through `createApp` on the embedded test database. 200 with the exact field set for an instance admin; 403 for a company owner who is not an instance admin, for an agent key, and unauthenticated; 404 and no collector when the flag is off; values move after agent requests, statements, a late run event and a missed renewal; the body has no IDs, keys or SQL. - `cd server && npx vitest run src/services/native-runtime/native-execution-lease.test.ts src/services/native-runtime/paperclip-control-plane-port.test.ts src/__tests__/api-server-timing.test.ts`: the lease and run-event hooks (red/green checked for the lease hook by removing the call). - `cd packages/db && npx vitest run src/lazy-transaction-queries.test.ts src/client-options.test.ts`: the counter counts statements inside transactions and savepoints. - `node --test scripts/scale-ramp/scale-ramp.test.ts` and `tsc -p scripts/scale-ramp/tsconfig.json`: 38 pass, including the fixture contract and the board-token header. - `node server/scripts/typecheck-layers.mjs` and `tsc --noEmit` in `packages/db`: pass. Overhead, measured on a shared, loaded laptop (load average 8 to 20), median with min and max: | Measure | Off | On | | --- | --- | --- | | Logger call per statement (5M calls x 7) | 0.28 ns (0.24 to 0.97) | 3.36 ns (3.32 to 6.16) | | CPU per `select 1` statement, embedded Postgres (3,000 x 10) | 28.7 us (20.3 to 42.4) | 32.3 us (20.7 to 45.5) | | CPU per HTTP request, client and server in one process (2,000 x 6) | 67.0 us (40.4 to 107.4) | 64.2 us (40.9 to 83.3) | | Idle process CPU, 5 s x 3 | 0.01% of a core (0.00 to 2.21) | 0.08% (0.05 to 0.09) | | Record one run event (Date.parse + histogram) | - | 148 ns (135 to 188) | | One snapshot | - | 55 us | The statement and request differences are inside the run-to-run spread. The direct cost is about 3 ns per statement, against about 30 us of CPU per statement. ## Risks - Low risk. Off by default; off, only the statement counter runs, and it costs one increment more than Drizzle's no-op logger. - `mainThreadCpu` assumes Linux USER_HZ of 100 and is null on other platforms. The harness then uses `eventLoopUtilization`. - Run event lateness compares the runner's clock with the server's clock, so clock skew adds to it. Negative values count as zero. - With several processes, each request reaches one process. The harness keeps the highest gauge and sums counter growth per process, but a process it never samples is not seen. - The statement counter does not count BEGIN/COMMIT or raw postgres.js calls. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file edits, local test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
45125f7748 |
test(server): stop the accounting identity child from timing out on loaded runners (#21712)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The server CI runs the general test suite in 12 shards on shared runners > - One accounting test fails at random on unrelated pull requests and passes on re-run > - Each failure costs a full re-run of the shard and the `verify` job, and it teaches people to re-run instead of read > - The test spawns two helper processes that load about 1,400 TypeScript modules under a 5 second kill timer > - This pull request makes the helper processes plain `node` processes, so they do not load the server module graph > - The benefit is that the test no longer depends on runner load, and it still uses real exited processes ## Linked Issues or Issue Description **What happened?** `server/src/services/run-cost-accounting.test.ts > run cost accounting work > normal accounting Retry reconstructs an uncommitted legacy receipt after its real controller and child exited` failed in "General tests (server (5/12))" three times on 2026-11-15 (UTC), each with the retry also failing: - PR #21687, head `2172a5b26a`, job 132643921517 - PR #21687, head `6a1d2fb0ef`, job 132650244103 - PR #21678, head `fc0e96f23c`, job 132663296537 Every failure has the same assertion: `AssertionError: {"exitCode":null,"signal":"SIGTERM","timedOut":true,...,"stderrTail":"accounting-identity:program-entered\naccounting-identity:module-imported\n"}: expected null to be +0` In five of the six failed attempts, the child stopped after `module-imported` and before `birth-read`. In one attempt, the child printed its result but the 5 second timer fired before it exited. **Expected behavior** The test passes on a loaded runner. The helper processes are not under test. Only the accounting Retry decision is under test. **Steps to reproduce** 1. Slow each module load in the child by 8 ms with an `--import` hook in `NODE_OPTIONS`. The test passes `NODE_OPTIONS` to the child. This simulates a loaded runner. 2. Run `npx vitest run src/services/run-cost-accounting.test.ts -t "normal accounting Retry"` in `server/`. 3. On `master` the test fails with the same `timedOut: true` assertion as CI. **Root cause** The child ran `node --import tsx/esm -e "await import('controller-process-identity.ts'); await currentProcessStartedAtMs(); ..."`. That import pulls in `@paperclipai/shared` and `@paperclipai/db` source through tsx, which is 737 modules. On Linux, `readProcessStartedAt` also dynamically imports the full runner package shim (`server/src/vendor/paperclip-runner/index.ts`) to call the 30-line `/proc` reader. That adds 652 more modules. tsx transpiles all of them in a cold process under a 5 second `timeoutSec`. On a shared CI runner this crosses 5 seconds. **Production impact** None. In the built server, the runner package is already loaded at boot, and `currentProcessStartedAtMs()` on Linux reads `/proc` once per process and caches the result. The cost exists only in a fresh tsx process that imports source. ## What Changed - The accounting identity helper child is now `node -e` with no tsx loader and no module imports. It installs a `SIGUSR2` handler, prints `ready`, and waits. - When the test sees `ready`, it reads the child start time with the product function `observedProcessStartedAt(pid)` and then sends `SIGUSR2`. The child exits with code 0. - The test builds the recorded `ControllerProcess` (random boot id, child pid, `currentProcessSpaceHash()`, observed start time). These are the same values the child computed for itself before. The child process is still real, has its own process group, and has exited before the Retry checks run. - No retries, skips or timeout changes. The 5 second guard now covers a plain `node` start, which takes about 40 ms. ## Verification - Red on `master` with the slow-load hook: `Tests 1 failed`, `timedOut: true`, the same assertion as CI. - Green with this change and the same hook: `Tests 1 passed`. - Full file without the hook: `npx vitest run src/services/run-cost-accounting.test.ts` passes 34/34. - Helper loop, 30 runs per variant, macOS, 18 cores with 36 busy-loop processes (load average about 56): - old child (macOS path): 29/30 exited in 5 s, p50 2,020 ms, p90 2,234 ms, max 5,172 ms - old child plus the runner import (as on Linux CI): 29/30, p50 3,663 ms, p90 4,411 ms, max 5,023 ms - new child: 30/30, p50 38 ms, p90 43 ms, max 46 ms - The full test under the same stress: 6/6 before and 6/6 after on macOS. macOS uses the `ps` path and does not load the runner package, so this loop does not show the Linux failure. The helper loop and the slow-load repro do show it. - `npx tsc --noEmit -p server/tsconfig.json` passes. - Other tests with the same pattern: `native-controller-source-digest.test.ts` also spawns a tsx child, but with a 60 second timeout. It is not the same cause, so this pull request does not change it. ## Risks - Low risk. Only one test changes. No product code changes. - The test no longer checks that a separate process computes its own identity stamp. That was not the purpose of this test. Other tests cover `currentControllerProcessStamp()`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edit, GitHub CLI). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
8adf0a463a |
perf(db): build the agent request authentication statements once per client (#21707)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - One Node process controls every run of a workspace. At 1,000
concurrent agents, main-thread CPU for each database statement is the
scarce resource.
> - A CPU profile of the hosted server shows the cost is per statement:
building SQL text in Drizzle costs about as much as parsing the result
in postgres.js.
> - Every authenticated agent API request (about 0.3 per agent per
second, plus tool calls) runs three fixed statements in
`actorMiddleware`: the agent row, the run authentication projection, and
the responsible user's company memberships. The server built the SQL
text of all three again for every request.
> - A local micro-benchmark shows that building the text is 15 to 40 µs
of CPU per statement for these shapes, and that a reused prepared query
costs the same as a raw driver call. Drizzle's `tracing.ts` wrapper is a
no-op (0.1 µs).
> - This pull request prepares the three statements once per database
client with placeholders, and executes them with the request's values.
The driver receives the same SQL text and parameters as before.
> - The benefit is about 40% less main-thread CPU for each authenticated
agent request in a local benchmark.
## Linked Issues or Issue Description
**What existing behavior does this improve?**
The CPU cost of agent JWT and agent key authentication on every agent
API request.
**Subsystem affected**
Server request authentication (`server/src/middleware/auth.ts`,
`server/src/middleware/agent-run-authentication.ts`).
**Current behavior**
For every agent request, `actorMiddleware` builds three Drizzle queries
from scratch: `agents` by id, the `heartbeat_runs` authentication
projection (with `jsonb_build_object` and two `run_identity_contexts`
subqueries), and the `company_memberships` read with an `exists` on
`auth_users`. Building and preparing them costs 14, 38 and 40 µs of CPU
(median of 7 passes of 5,000 builds).
**Proposed behavior**
Build each statement one time for each database client, with
`sql.placeholder` values. Execute the prepared query with the request's
values. The SQL text, the parameters, and the returned rows stay the
same.
**Reason and benefit**
At 1,000 agents, agent API traffic is several hundred authenticated
requests per second on one main thread. Measured CPU per authenticated
request (local, embedded PostgreSQL, `process.cpuUsage()`, 2,000
requests per pass):
| Run (interleaved) | Before: median (min–max) µs | After: median
(min–max) µs | Load average |
| --- | --- | --- | --- |
| 1 | 436 (363–599) | 200 (183–224) | 10.8 / 10.1 |
| 2 | 489 (430–535) | 312 (212–392) | 11.6 / 13.4 |
| 3 | 461 (442–562) | 281 (247–409) | 13.1 / 12.9 |
| separate | 383 (322–810), 301 (268–476) | 169 (156–439), 193 (174–215)
| 9.5 / 9.0 |
The machine is shared (18 cores), so the spread is wide. Every pair
shows the after run lower. The drop is larger than the build cost alone,
because the request also allocates less (fewer builder objects to
collect).
**Breaking changes**
None. The statements, their parameters and the result shapes do not
change.
**Additional context**
Per-statement cost attribution from the new
`packages/db/scripts/benchmark-statement-cpu.ts` (CPU µs per statement,
median of 5 passes of 2,000, two runs, concurrency 1, load average
20–26):
| Statement | Drizzle builder | Build only (`toSQL`) | Drizzle
`.prepare()` reused | `unsafe(text, params)` | `unsafe(..., {prepare:
true})` |
| --- | --- | --- | --- | --- | --- |
| `issues` parent by id | 105 / 122 | 17 / 18 | 59 / 75 | 57 / 63 | 30 /
33 |
| `issues` full row by id | 277 / 326 | 136 / 153 | 96 / 81 | 97 / 77 |
48 / 33 |
| `company_memberships` by principal | 128 / 103 | 35 / 33 | 71 / 66 |
80 / 63 | 37 / 32 |
| join with an in-list of 10 | 169 / 173 | 57 / 57 | 83 / 88 | 109 / 80
| 42 / 40 |
| update by id | 110 / 106 | 25 / 26 | 75 / 80 | 76 / 73 | 39 / 40 |
| insert `activity_log` | 116 / 125 | 34 / 35 | 65 / 57 | 64 / 55 | 41 /
30 |
Drizzle's no-op tracer (`startActiveSpan`) measured 0.1 µs per call.
## What Changed
- `server/src/services/prepared-statements.ts`: add
`preparedStatement(prepare)`, the single-statement form of the existing
`preparedStatements` helper (one prepared query for each `Db`, held in a
`WeakMap`).
- `server/src/middleware/agent-run-authentication.ts`: split the query
into `agentRunAuthenticationQuery(db, binding)`;
`loadAgentRunAuthentication` executes the query prepared once with
placeholders.
- `server/src/middleware/auth.ts`: add `agentAuthenticationQuery` and
`responsibleUserMembershipsQuery`; the JWT path, the agent key path and
`loadResponsibleUserMemberships` execute their prepared forms.
- `server/src/__tests__/agent-auth-prepared-statements.test.ts` (new):
proves that each prepared statement sends the same SQL text and
parameters and returns strictly equal rows, and that repeated agent
requests do not build the statements again.
- `server/src/__tests__/agent-auth-middleware.test.ts`: the test
double's select chain now supports `.prepare().execute()`.
- `server/src/__tests__/prepared-statements.test.ts`: covers
`preparedStatement`.
- `packages/db/scripts/benchmark-statement-cpu.ts` (new): CPU per
statement for the builder, build only, a reused prepared query, `unsafe`
with and without `prepare: true`, and the no-op tracer. Run with `cd
packages/db && ITERATIONS=2000 PASSES=5 node --import tsx
scripts/benchmark-statement-cpu.ts` (`CONCURRENCY`, `ONLY`, `VARIANTS`
are optional).
- `scripts/benchmark-agent-request-auth-cpu.ts` (new): CPU per
authenticated agent JWT request through `actorMiddleware`. Run with
`pnpm --filter @paperclipai/server exec tsx
../scripts/benchmark-agent-request-auth-cpu.ts`.
Both benchmarks start their own embedded PostgreSQL in a temporary
directory.
## Verification
- Red/green: the "builds the request authentication statements once per
database client" test failed on `origin/master` (`select` called 12
times in 3 requests) and passes with this change.
- `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/agent-auth-prepared-statements.test.ts
src/__tests__/prepared-statements.test.ts
src/__tests__/agent-auth-middleware.test.ts
src/__tests__/agent-run-authentication.test.ts
src/__tests__/agent-jwt-request-authentication.test.ts
src/__tests__/auth-session-route.test.ts
src/middleware/cloud-tenant-actor.test.ts
src/__tests__/local-agent-process-boundary.test.ts
src/__tests__/liveness.test.ts src/__tests__/inbox-mine-routes.test.ts
src/__tests__/board-bootstrap-routes.test.ts
src/__tests__/agent-auth-identity-transfer.test.ts
src/__tests__/prepared-statements-driver.test.ts
src/services/fresh-continuation-fallback.test.ts`: all pass.
- CPU before and after: the tables above.
## Risks
- Low risk. The prepared query uses Drizzle's own `PgPreparedQuery`, the
same execute path that the builder uses. The test pins the SQL text, the
parameters and the rows against the builder.
- Drizzle's `.prepare(name)` on postgres.js does not send a statement
name to the database (see #21201). With `DATABASE_PREPARED_STATEMENTS`
unset, the driver sends the same unnamed statement as before, so
connection poolers see no change.
- A test double for `Db` must now support `.prepare().execute()` on
these selects. One test double changed.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use (shell, file edits, local test and benchmark runs).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
ea8c734cb9 |
fix(chat): park GitHub terminal responses that wait for a newer run or task change (#21705)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The GitHub chat channel posts one App-owned response comment per
accepted request. The terminal-response sweep
(`terminal-response-work.ts`) finds processed, non-final responses whose
task's latest run has ended, and asks the acknowledgement owner to post
the final text.
> - On paperclip.staging, 1,557 processed `github_response_comment` rows
are not final. None changed in the last 5 minutes, and the oldest dates
from 2026-10-28. The sweep reads all of them again on each company-wide
rebuild (every 30 s per company), with one page read plus three
statements per row. Two of those statements run at about 13/s each in
`pg_stat_statements`.
> - The sweep cannot make progress on these rows. Their task's latest
run ended without owning the request (a later request started it), or
the pause is already shown. Nothing that the decision reads changes
until a new run starts, the task status changes, or the row changes.
> - This pull request records that wait on the row and skips it on broad
rebuilds until one of those inputs changes.
> - The benefit is that an idle backlog costs one page read per rebuild
instead of about three statements per row, and no request loses a path
to settle.
## Linked Issues or Issue Description
No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/bug_report.yml`.
**What happened?**
On paperclip.staging (2026-11-15, about 01:45–02:25Z), two single-row
statements ran at nearly the same rate:
- `select id, run_id, company_id, status, responsible_user_id,
message_id, parent_context_id from run_identity_contexts where id = $1
and run_id = $2 and company_id = $3 and status = $4 limit $5` at
13.37/s.
- `select chat_conversations.issue_id, chat_endpoints.provider from
chat_message_links join chat_conversations … join chat_endpoints … where
chat_message_links.company_id = $3 and comment_id = $4 …` at 13.20/s.
They are `provenanceReads.context` and `provenanceReads.link` in
`run-origin.ts`. The terminal-response sweep reaches them through
`readGitHubResponseRun` for each row it loads.
The backlog of `github_response_comment` actions on staging:
- `processed`, `delivery_id` set, `result.final` not `true`: 1,557 rows
(the terminal sweep's set). None updated in the last 5 minutes. The
oldest `updated_at` is 2026-10-28.
- `result.code` of those rows: null 1,387, `acknowledgement_unconfirmed`
150, `acknowledgement_superseded` 20.
- The acknowledgement set (`received`, or `failed` and retryable) is
empty.
- Other rows: processed and final 6,645, failed and not retryable 265,
cancelled 116.
**Root cause**
`entry()` returns `null` ("nothing to post now") when
`githubResponseExecution` has no body. That happens when the task's
latest run has ended but does not own the request and the task is not
blocked. It also happens when the task is blocked and the row already
shows this pause (`pausedVersion` matches). The page query still selects
these rows, because the latest run has ended (or the task is blocked)
and the row is not final. Nothing records the decision, so each
company-wide rebuild reads every row again: `readGitHubResponseRun` (one
statement) plus `chatRunSourceMessageId` (the context and link reads)
for each row. The codes do not matter. `acknowledgement_unconfirmed` is
a retry code that a later successful acknowledgement keeps, and
`acknowledgement_superseded` marks a request whose shared PR comment has
a newer version. All three groups wait for the same thing: a newer run
that owns the request, or a task status change.
**Expected behavior**
A response that cannot progress costs nothing until the run, the task or
the response changes. A response that can progress settles as before.
**Steps to reproduce**
1. Create a GitHub endpoint and a task with several processed, non-final
response comments (codes null, `acknowledgement_unconfirmed`,
`acknowledgement_superseded`).
2. Give the task a newer ended run whose identity context names a later
request, so the run does not own the older requests.
3. Register the terminal-response sweep and send a company-wide hint
every 30 s.
4. On master, each rebuild reads every row again with about three
statements per row and writes nothing.
**Paperclip version or commit**
master at
|
||
|
|
82b243ab07 |
perf(server): bound dedicated database connections with a fair queue (#21693)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The server holds long advisory-lock transactions on dedicated
database connections (`withDedicatedDbConnection`): workspace source and
lifecycle locks, the skill cache lock, agent-directory operations,
status cards and others
> - `withDedicatedDbConnection` kept at most 4 idle connections, but it
had no limit on connections in use. A run start can hold two at once,
because the source lock nests the lifecycle lock
> - After a restart or deploy, a burst of run starts can open hundreds
of connections. That can exceed the database's `max_connections` at the
moment recovery needs the database most. The documented per-process
total (`DATABASE_POOL_MAX` + 2) was also wrong
> - This pull request bounds the dedicated connections in use with a
fair queue, keeps nested and in-transaction callers deadlock-free, and
states the real connection budget
> - The benefit is that a restart burst of 200 or 1,000 run starts
completes within the limit (at most 48 in use with the default limit of
24), where before most of the starts failed with "too many clients"
## Linked Issues or Issue Description
**What existing behavior does this improve?**
`withDedicatedDbConnection`, which gives long advisory-lock transactions
a connection outside the main pool.
**Subsystem affected**
Database client (`packages/db/src/client.ts`) and the connection-sizing
documentation.
**Current behavior**
Nothing bounds the dedicated connections in use; only 4 idle ones are
kept. A run start can hold two at once (the source lock nests the
lifecycle lock). A burst of run starts after a restart can open hundreds
of connections and exceed `max_connections`. The documented per-process
total (`DATABASE_POOL_MAX` + 2) is wrong.
**Proposed behavior**
A per-process limit on dedicated connections in use
(`DATABASE_DEDICATED_CONNECTION_MAX`, default 24) with a first-in,
first-out queue. A caller that already holds a dedicated connection, or
runs inside an open transaction, never waits, because its wait could
deadlock with the holder that blocks it. After
`DATABASE_DEDICATED_CONNECTION_WAIT_MS` (default 30 s), a queued caller
gets a connection over the limit, so a lock cycle that PostgreSQL cannot
see ends in a delay, not a hang. The documentation states the real
per-process budget.
**Reason and benefit**
At 50 agents, staging showed about 9 new connections per second from
this function. At 1,000 agents, a restart burst can take the database
down when recovery needs it most. With the default limit of 24, a burst
of 200 or 1,000 starts completes with at most 48 dedicated connections
in use.
**Additional context**
Refs the 1,000-concurrent-agents plan
(`doc/plans/2026-11-13-thousand-concurrent-agents.md`). Reusing the
outer session for a nested lock was considered. The locks are
transaction-level advisory locks (`pg_advisory_xact_lock`), so a nested
lock in the outer transaction would be held until the outer transaction
ends, which changes lock lifetimes and order.
## What Changed
- `packages/db/src/client.ts`: `withDedicatedDbConnection` admits
callers against a per-owner limit (`DATABASE_DEDICATED_CONNECTION_MAX`,
option `dedicatedConnectionMax`, default 24) with a first-in, first-out
queue. The slot is released after the connection is returned or closed.
- Nested callers (an active dedicated hold of the same owner in the
async context) and callers inside an open transaction of a client from
this process are admitted without a wait. The transaction marker is an
`AsyncLocalStorage` scope that ends when the transaction callback ends.
- A queued caller is admitted over the limit after
`DATABASE_DEDICATED_CONNECTION_WAIT_MS` (option
`dedicatedConnectionWaitMs`, default 30000).
- `dedicatedDbConnectionStats(db)` reports the limit, in-use, peak,
waiting, queued, over-limit and nested-over-limit counts.
- `PRIVATE_POOL_MAX_CONNECTIONS` names the private pool sizes that were
literals, and `databaseConnectionBudget()` returns the per-process worst
case: main pool + 9 private + 2 listeners + 2 x limit + 4 idle dedicated
= 73 with the defaults.
- `server/src/services/skill-package-store.ts`: an isolated package read
takes its package I/O permit before its dedicated connection, and takes
a second dedicated connection only for the cache trim after a publish.
Before, each of up to 32 queued readers held a dedicated connection
while it waited for one of 2 permits. `skillPackageStorageIoDepth()`
reports the queue for the store's test.
- `withDedicatedDbConnection(db, action, { admitWithoutWaiting: true })`
admits a caller at once. The skill package store passes it for the
dedicated connections it takes while it holds a package permit, so a
permit holder never waits for a slot that a cache-lock owner holds while
that owner waits for the permit. The overshoot is bounded by the permits
(2 storage, 2 response).
- `doc/DATABASE.md` and `docs/deploy/database.md`: document the
admission rules, the two new variables and the real connection budget,
which replaces "`DATABASE_POOL_MAX` + 2".
## Verification
- New `packages/db/src/dedicated-connection-admission.test.ts` (embedded
PostgreSQL, `max_connections` 100). A restart burst with the default
limit: each start holds a source advisory lock on a dedicated connection
and nests a lifecycle lock on a second one, 40 ms in all.
- Before (same burst, scratch test on master): 200 starts gave 151
failures ("sorry, too many clients already") and 99 dedicated sessions
open. 1,000 starts gave 823 failures and 99 sessions.
- After: 200 starts gave 0 failures, 45 sessions, peak in use 48, queue
wait p50 541 ms / p99 815 ms. 1,000 starts gave 0 failures, 40 sessions,
peak in use 48, wait p50 1,767 ms / p99 3,422 ms. The wait is the queue
of this synthetic burst of 40 ms holds. Before, the same callers waited
about as long for connection setup, and most then failed.
- Steady state: 20 concurrent holders under the limit are never queued
(`queued: 0`).
- Nested acquisition: with a limit of 2, two holders each take a nested
connection while a third caller is queued. All finish (no deadlock); the
queued caller runs after them.
- In-transaction caller: with a limit of 1, a pooled transaction holds a
row lock and then asks for a dedicated connection, while the one
dedicated holder waits for that row lock. The caller is admitted at once
and both finish. A hard limit would deadlock here.
- Resource holder: with a limit of 1, the slot holder waits for a
resource that a second caller holds. The second caller, with
`admitWithoutWaiting`, is admitted at once and both finish.
- Wait bound: with a limit of 1 and a 100 ms bound, a queued caller is
admitted over the limit after the bound.
- Budget: the sum of the actual `max` of every private pool equals
`databaseConnectionBudget().privatePools`, and the default total is 73.
- `skill-package-store.test.ts` ("does not load queued legacy bodies for
files reads"): 32 readers queue for package I/O while only the 2 active
readers hold dedicated connections, and the 33rd reader still gets
"busy" at once. With the old order and a limit of 24, CI showed only 22
readers reaching that queue. `skill-package-store`,
`skill-version-cache`, `company-skills-empty-runtime`,
`runtime-skill-preservation`, `company-skills-runtime-concurrency` and
`company-skills-service` pass locally. Two `company-skills.test.ts`
workspace-discovery tests fail locally on master too (same result with
this change), so they are not caused by it.
- `packages/db`: client, lazy-transaction, prepared-query,
response-capture, work-signal and dedicated-pool suites, 134 passed.
- `server`: skill-version-cache, status-card-evaluation-work,
git-workspace-expiration, execution-workspace-runtime-control-conflict,
agent-instruction-working-copies, company-skills-empty-runtime and
decisions-service suites: 413 passed. In one combined run under machine
load, git-workspace-expiration failed at file level with all its tests
passing; alone it passed 96 of 96.
- `npx tsc --noEmit` in `packages/db`: passes.
## Risks
- A caller that holds an application-level lock (for example a
filesystem lock) and then waits in the queue can block a dedicated
holder that needs that lock. The 30 s wait bound ends such a cycle with
a delay. The over-limit counter shows it.
- Nested and in-transaction callers do not wait, so the in-use count can
reach twice the limit with one level of nesting. The documented budget
counts this.
- In a large restart burst, run starts now queue for a slot instead of
opening new connections. Steady state (about 10 to 20 dedicated
connections at 1,000 runs) stays under the default limit of 24, so a
run's first turn does not wait there.
- The checkpoint pool (3 connections) can limit throughput at 1,000 runs
on a hosted database. This pull request does not change it; it needs an
owner decision on the hosted connection budget.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
and code execution.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
9c230ace94 |
fix(heartbeat): settle deferred chat wakes that can never resume instead of retrying every pass (#21678)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The heartbeat deferred-receipt worker resumes wakes that waited
behind a running task execution
> - A chat message that arrives while its task runs is saved as a
deferred chat wake
> - When the task finishes (done or cancelled) first, the saved wake can
never resume, but the sweep retried it forever and touched `updated_at`
on every pass
> - An idle instance did one write and one failed resume about 4 times a
second, and the log line did not say why
> - This pull request gives each such wake one recorded outcome, and
makes every other failure wait on a stored, spaced, bounded next attempt
> - The benefit is an idle database stays idle, and a person sees the
existing "not started" result instead of silence
## Linked Issues or Issue Description
**What happened?**
The stranded deferred-queue sweep in
`server/src/services/heartbeat/queue.ts` selects
`deferred_issue_execution` chat-inbound wakes. On every pass it runs
`update agent_wakeup_requests set updated_at = now()` for each wake.
Then `resumeDeferredChatWake` fails because the task is done or
cancelled, so the chat authority denies the work start
(`task_assignment`). Nothing records the failure, so the next pass
repeats it.
**Steps to reproduce**
1. Save a chat message on a task while a run is active, so the message
becomes a deferred chat wake.
2. Let the run finish and mark the task done.
3. Leave the instance idle. The log shows "saved chat source could not
resume after recovery" again and again for the same request ids.
**Expected behavior**
A wake that can never resume gets one terminal outcome with a stored
reason. A wake that can resume later waits on a stored time. An idle
instance runs no statements for these wakes.
**Paperclip version or commit**
master at
|
||
|
|
11b09b7234 |
refactor: remove doc/execution-semantics.md and its references (#21699)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Contributors and agents read the docs under `doc/` for background on run execution > - `doc/execution-semantics.md` grew to 6,310 lines of per-PR notes > - It was hard to read and it drifted from the code > - The code and tests are the source of truth for execution behavior > - This pull request removes the file and every reference to it > - The benefit is one less stale document that readers can mistake for a requirement ## Linked Issues or Issue Description **Issue type** Outdated or unclear documentation. **Where is the issue?** `doc/execution-semantics.md`, and the docs, rules, skills, maps, code comments and log text that link to it. **What's wrong?** The file is 6,310 lines of notes added one PR at a time. Much of it repeats or contradicts the code. Readers and agents treat it as a contract, but the code and tests decide the behavior. **Suggested fix** Delete the file. Remove each link to it. Keep the documents that link to it. Point at code where a pointer is still useful. Do not move the content to another doc. Refs #10664 (an open docs PR that edits this file; it becomes obsolete). ## What Changed - Delete `doc/execution-semantics.md`. - Remove links and pointers in `doc/PRODUCT.md`, `doc/LOW-TRUST-PRESETS.md`, `doc/TASK-WATCHDOG.md`, `doc/composer-stop.md`, `doc/native-workspace-finalization-recovery.md`, `doc/run-log-events.md`, `doc/runner-internals.md` and `doc/workspace-repositories.md`. - `doc/TASK-WATCHDOG.md` now points at `server/src/modules/active-run-watchdog/` for the silent active-run watchdog. - Remove the link in `rules/upgrade-safe-run-state.mdc` and `maps/feature-map/recovery.md`. - Update the skills `diagnose-why-work-stopped`, `fix-failed-paperclip-run` and `terminal-bench-loop`. The `fix-failed-paperclip-run` skill now points at `server/src/services/heartbeat/` and `server/src/services/recovery/`. - Remove the file from the source anchors in `.devin/wiki.json`. - Remove the doc name from four code comments in `server/` and `ui/`. - The failed-attachment log message no longer ends with a link to the removed doc. Two tests now check the message text instead of the link. - Leave two dated plan files under `doc/plans/` unchanged. They are history. ## Verification - `git grep -i execution-semantics` finds no hits outside `doc/plans/`. - Every relative Markdown link in the changed files resolves. - `node scripts/check-maps.mjs` passes. - `node --test .github/scripts/tests/*.test.mjs` passes (808 tests). - `pnpm verify` was not run locally. The worktree has no `node_modules`. CI runs the changed Vitest files. ## Risks - Low risk. Most changes are docs and comments. - One runtime log line is shorter. It no longer has a link. No logic changes. - Open PRs that edit the deleted file, such as #10664, will conflict. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and shell access. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
adc3356ad3 |
fix(native): retry a coordinator claim that loses a deadlock instead of failing the run (#21703)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native (Paperclip Runner) run claims its `native_run_finalizations` coordinator before it starts any provider work. The claim is one transaction. It locks the coordinator row, then the run row. > - A writer that holds the run row and then waits for the coordinator row closes a lock cycle. PostgreSQL then aborts the claim with a deadlock (`40P01`), because the claim waited first. > - The aborted claim rolls back whole and starts no provider work. But the raw database error failed the run before its first turn. > - The failure path then preserved the prepared workspace. That stamped the unclaimed coordinator `native_continuation_requires_reconciliation`, and `failRun` used it as the run's error code. The task waited for reconciliation. > - A transient database fault must never end a run like this. The target rate of runs that need Retry is 0%. > - This pull request retries the claim transaction on transient lock contention with the shared bounded helper. > - The benefit is that a fresh run that loses a lock race at its claim continues to its provider turn by itself. ## Linked Issues or Issue Description **What happened?** On a hosted QA stack, a fresh native assignment run (not a retry) failed after about 50 s. The run status was `failed`. The error code was `native_continuation_requires_reconciliation`. The error text was `Database query failed (40P01)`. The stderr was empty. The task needed reconciliation. **Steps to reproduce** 1. Prepare a native run with an unclaimed coordinator (`observed`, attempt 0), as runtime selection leaves it. 2. In another transaction, lock the run row. 3. Start the native session. Its claim locks the coordinator and waits for the run row. 4. In the other transaction, lock the coordinator row. 5. PostgreSQL aborts the claim with `40P01`, and `executePaperclipNativeSession` rejects with that error. **Expected behavior** The claim retries after the competing transaction commits. The run reaches its provider turn. **Paperclip version or commit** `master` at |
||
|
|
c38a471042 |
perf(server): download stale agent files before the agent row lock (#21694)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Durable agent files live in object storage. Each server keeps a local copy (the projection), marked with the head generation it matches > - When the projection is stale, the next locked read or write restores it: it downloads every file from object storage. This happens for the first run of each agent on a new server (every deploy on hosted ephemeral servers) and after another server wrote > - The download ran inside the transaction that holds the `agents` row lock (`FOR UPDATE`), on the main pool for the run-start read (`readCommittedForRuntime`) and on the 2-connection file-storage pool for the store. A download can take up to 30 s > - After a deploy at 1,000 agents, this is one long row lock and one pinned connection per agent, all at once, and every writer of that agent waits behind it > - This pull request downloads the files outside the transaction and takes the lock again only to swap them in > - The benefit is that the row lock is held for milliseconds instead of the download time (0 ms observed instead of 321 ms with a 300 ms download in the test), and writers do not wait for a download ## Linked Issues or Issue Description **What existing behavior does this improve?** The restore of a stale local copy of an agent's durable files (the first run of each agent on a new server, or after another server wrote). **Subsystem affected** Server: durable agent files (`agent-file-durability.ts`, `agent-file-store.ts`, `agent-instruction-revisions.ts`). **Current behavior** The restore downloads every file from object storage inside the transaction that holds the `agents` row lock (`FOR UPDATE`) and a pooled connection. A download can take up to 30 s, and every writer of that agent waits behind it. **Proposed behavior** Under the lock, detect that a download is needed and roll back. Download the head's files, verified by digest, into a private staging directory with no lock. Run the transaction again, and swap the staged files in only when the head generation is unchanged. After two changed heads, restore under the lock as before. **Reason and benefit** After a deploy at 1,000 agents, every agent's first run restores at once. The row lock and connection are now held for milliseconds instead of the download time, and writers do not wait for a download. **Additional context** Refs the 1,000-concurrent-agents plan (`doc/plans/2026-11-13-thousand-concurrent-agents.md`). The save path had the same problem for uploads and was fixed earlier with uploads before the lock (`uploadAgentFilesBeforeCommit`); this is the restore side. Reading the head before every transaction was considered; it adds a query to every call, while the deferral costs nothing when the projection is current. ## What Changed - `agent-file-durability.ts`: the download loop moves into `downloadAgentFiles`, unchanged. `withAgentFilesRestoredBeforeLock(work)` runs `work` in a scope. Inside the scope, the first restore of an attempt that needs a download throws a private deferral error, which rolls back the transaction. The wrapper stages the download in `.agent-files-<uuid>` next to the root and runs `work` again. `restoreDurableAgentFiles` uses a staged directory only when its generation equals the head it reads under the lock. Staged directories are removed when the scope ends. - Only the first restore of each attempt can defer, so nested restores of other agents keep their old behavior. Outside the scope, the behavior is unchanged. - `agent-file-store.ts` (`locked`, so every store read, write, apply and working-copy preparation) and `agent-instruction-revisions.ts` (`readCommittedForRuntime`, `readCurrent`, `materializeCurrent`) run their transactions in the scope. ## Verification - New test `server/src/__tests__/agent-file-restore-locking.test.ts` on embedded PostgreSQL, with an object-storage fixture: - Hold time, with a 300 ms download, through `readCommittedForRuntime` (the run-start read). Before: the download ran with the agent row locked, and the row was locked 321 ms of 322 ms. After: the download ran with the row unlocked, and a sampler saw the row locked for 0 ms. - Concurrency: a board write of the same file runs while the reader's download is blocked. Before: the writer waited for the reader's download (5 s timeout). After: the writer commits a new head during the download. The reader then discards its stale staged copy and returns the writer's content. The projection is marked at the writer's generation, and no staging directory is left. - Failure: when object storage fails, the read fails once with the storage error, after one attempt made without the row lock, and leaves no staging directory. - Progress: when the head generation changes during each of two downloads, the third attempt restores under the lock as before (3 downloads), and the projection matches the final head. - `npx vitest run` for agent-file-checkpoint-locking, agent-file-checkpoints, agent-file-durability, agent-file-restore-locking, agent-instruction-revisions, agent-instruction-working-copies, agent-instructions-service, agent-directory-working-copies and agent-instructions-routes: 287 passed, 1 skipped. - `npx tsc --noEmit -p server/tsconfig.json`: passes. ## Risks - A deferred attempt runs the transaction body again. The deferral happens at the first restore, before the body writes files, and the transaction rollback undoes database work, so the second attempt starts from the same state. - A staged download uses disk next to the agent root until the scope ends, the same as the temporary directory of the old download. - If the head changes during every download, the third attempt downloads under the lock as before. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
cb9fe3f72d |
perf(server): capture a remote agent-file checkpoint before the directory lock (#21690)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A warm remote run saves its agent files at the end of every turn (`checkpointWarm`) > - On hosted (ephemeral) servers, this save holds the cross-controller `agent-directory` advisory lock in an open transaction on a dedicated database connection > - The lock was held across the full remote capture: an asset upload, the checkpoint script, the download and the scratch cleanup in the sandbox. Each step is a sandbox round trip of hundreds of milliseconds or more > - At 1,000 concurrent agents, connection-seconds are the scarce resource, and this hold occurs on every turn of every warm remote run. The lock is per agent, so a second run of the same agent got "Agent directory operation is already active" for the whole capture > - This pull request captures the remote files before the lock, and takes the lock only to commit > - The benefit is that the lock and its connection are held for the commit only (85 ms instead of 2.5 s in the test with a 200 ms sandbox), and no sandbox call runs under the lock ## Linked Issues or Issue Description **What existing behavior does this improve?** The per-turn save of agent files for a warm remote run (`checkpointWarm`) on hosted servers. **Subsystem affected** Server: agent directory working copies (`server/src/services/agent-directory-working-copies.ts`). **Current behavior** The save holds the per-agent `agent-directory` advisory lock, an open transaction and a dedicated database connection across every sandbox round trip of the remote capture. A second run of the same agent gets "Agent directory operation is already active" for that whole time. **Proposed behavior** Capture the remote files with no lock. Under the lock, use that capture only when the working-copy row is unchanged. Otherwise discard it and capture again under the lock, as before. **Reason and benefit** At 1,000 concurrent agents, connection-seconds are the scarce resource, and this hold occurs on every turn of every warm remote run. The lock is now held for the commit only. **Additional context** Refs the 1,000-concurrent-agents plan (`doc/plans/2026-11-13-thousand-concurrent-agents.md`). A lease row in place of the advisory lock was considered; it is a larger change, and the optimistic capture removes the slow part without a new protocol. ## What Changed - `agent-directory-working-copies.ts`: `checkpointWarm` captures the remote checkpoint before `serial()` for a disposable remote copy (hosted servers only), with the same inputs that the locked capture uses. - Under the lock, `checkpoint()` reuses that capture on its first attempt only when the current row equals the row the capture used (`adoptionAuthority`, which excludes only `updatedAt` and `nextAttemptAt`) and the target is the same. If the row changed, it captures again under the lock, as before. - If the early capture fails, the code logs a warning and captures under the lock, as before. The early capture's scratch is removed after the lock is released. - Local copies and non-hosted servers are unchanged. ## Verification - New test `server/src/__tests__/agent-directory-remote-checkpoint-lock.test.ts` on embedded PostgreSQL, with the real checkpoint transport and a fixture sandbox runner that takes 200 ms per call: - Before: 11 of 11 sandbox calls ran while the `agent-directory` advisory lock was held. The lock was held 2,482 ms of a 2,515 ms checkpoint. - After: 0 of 11 sandbox calls ran under the lock. The lock was held 85 ms of 2,863 ms. - Concurrency: a competing collector (`collectStopped` from a second service instance) runs while the early capture is blocked in the sandbox. Before: the collector failed with "Agent directory operation is already active". After: the collector saves and releases the copy, and the checkpoint discards its early capture and returns the collector's row unchanged. The canonical file has the final turn's content once, and no scratch directory is left. - `npx vitest run src/__tests__/agent-directory-working-copies.test.ts src/__tests__/agent-file-checkpoint-locking.test.ts src/__tests__/heartbeat-agent-directory-busy.test.ts src/__tests__/agent-directory-lock-crash-point.test.ts src/__tests__/agent-instruction-working-copies.test.ts`: 189 passed, 1 skipped. - `npx tsc --noEmit -p server/tsconfig.json`: passes. ## Risks - The capture now runs while another controller can operate on the same copy. The capture only reads the remote directory into private scratch. The commit uses it only when the row is unchanged, so the result is the same as one of the orders the lock allowed before: the checkpoint first, or the other operation first and then the old locked checkpoint. - The agent can change remote files between the capture and the commit. That was also true before, because the agent is not frozen during a checkpoint. - Changed bytes are still uploaded to object storage inside the lock (before the agent row lock). Moving that upload out is a possible follow-up. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
d6acbf50e6 |
refactor: cut DATABASE.md to setup and schema-change rules (#21700)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - `AGENTS.md` tells every contributor and agent to read `doc/DATABASE.md` before a schema or migration change > - That file grew to 2,611 lines. About 1,950 of them were notes on the tables of one feature, added one pull request at a time > - The setup steps and the migration rules started at line 792, below those notes > - This pull request cuts `doc/DATABASE.md` to 377 lines: setup, and the rules that apply to every schema change > - The benefit is a guide that a person or an agent can read in full before a schema change ## Linked Issues or Issue Description Refs #21670 (the same cleanup for `doc/DEVELOPING.md`). **Issue type** Outdated or confusing documentation. **Where is the issue?** `doc/DATABASE.md` **What's wrong?** The file was 2,611 lines. It opened with a Better Auth upgrade note and 492 lines on task query performance. The schema change workflow started at line 1367. **Suggested fix** Keep database setup and the general rules in `doc/DATABASE.md`. Remove the notes on individual tables. The schema and its tests describe those. ## What Changed - `doc/DATABASE.md` is now 377 lines. It keeps the three database modes, client tuning, the schema change workflow, migration compatibility, the migration authoring checklist, migration snapshots, the rule for retries after a lost connection, transaction-aware work signals, plugin namespaces, and backups. The wording of the kept text is unchanged. - New `doc/database-client-internals.md` holds the response-size diagnostic, the notes on the postgres.js patch, and the transaction memory note. The text is moved without wording changes. - "Experimental remote workspace snapshots" moved to `doc/workspace-storage.md`, which links to it. - "Execution admission accounting" moved to `server/src/modules/README.md`, which links to it. - Deleted about 1,950 lines of notes on one feature each. No other file linked to them. The deleted sections are: Better Auth 1.7.16 upgrade, Durable native provider backups, Workspace cohort admission proofs, Original local Git sources for archived workspaces, Proactive agent chat messages, Persistent case handoffs, Task query performance, Tool profile reads, Execution identity row locks, Remote workspace row locks, Managed adapter settings, Cloud configuration migration receipt, Cloud runtime identity singleton, Unclaimed Cloud warm standby, Native finalization history and idle sleep, Feedback history and idle sleep, Resource membership tables, Decision training snapshot retention, Decision queues and triage provenance, Native runner persistence, Chat communication snapshots, Telegram private draft identities, Attachment upload provenance, Question-response delivery receipts, Secret storage, Persistent agent conversations, Resource lifecycle events, Legacy controller ownership, Agent file persistence and legacy revisions, Large API response snapshots, Runner API response reservations, Internal agent commentary, Agent identity keys and backups, Slack app registration, Slack manager workspace grants, Installed skill packages, Project repository skill synchronization, Task workspace records, Execution recovery query performance, Agent runtime synchronization journal, Retired Pipelines storage, Prompt injection classification, Deferred conversation requests, GitHub inherited review-wait reads, Sandbox creation ownership, Object-backed run evidence, Cloud runtime configuration singleton, Independent accounting and budget evaluation, Standard workspace and Classic opt-in, Project environment rolling compatibility. - Also deleted from kept sections: the subscription-table note and the issue-reference backfill note in the embedded mode section, the HTTP access log note and the per-endpoint retry notes in the connection loss section, and the notes on migrations `0355`, `0313`, and `0314` in the authoring checklist. - `AGENTS.md` now states what `doc/DATABASE.md` is for. ## Verification - A script built the new files by line range from the old file. It asserts the old line count and the first line of each range. - A link check on every changed Markdown file, and on the four files that link to a `DATABASE.md` anchor, found 0 missing files and 0 missing anchors. - `node scripts/check-maps.mjs` passes. - `node --test .github/scripts/tests/*.test.mjs`: 808 pass, 0 fail. - `pnpm verify` was not run. This worktree has no installed dependencies, and the change has no runtime code. ## Risks - The deleted notes were not all read in full before deletion. The decision was by section: a section about one feature's tables, with no inbound link, was deleted. The text stays in git history. - Two deleted sections describe procedures for an operator: the duplicate-account check before the Better Auth 1.7.16 migration, and the secret storage tables. Restore them into a focused doc if operators still need them. - No runtime behavior changes. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`), Claude Code desktop, with tool use. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [ ] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |