mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-11 05:11:49 +01:00
693de27c906509fd141fdabefa2d8b5accfdc5bc
7294
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 DottaandClaude Opus 5.5
|
693de27c90 |
fix(storage): wait out expired or rotated S3 credentials and re-resolve the key (#20456)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - In cloud mode, run logs, provider traces, attachments and remote workspace snapshots go to S3 > - The server uses the default AWS credential chain, and real keys expire or rotate while runs are in flight > - #20299 made runs wait out a storage outage, but a 403 `ExpiredToken`, `InvalidAccessKeyId` or `SignatureDoesNotMatch` was a permanent `adapter_failed` > - The SDK also caches a key until its stated expiration (forever for a key with none), so a rotated key was never picked up > - This pull request treats credential rejections as a storage wait and re-resolves the key from the credential chain after a rejection > - The benefit is that runs survive key expiry and rotation without failed attempts or a manual Retry ## Linked Issues or Issue Description Refs #20299 (storage outage wait). Harness that found this: #20455. **What existing behavior does this improve?** Runs wait out S3 outages (5xx, timeouts, resets) but fail at once when the store rejects the server's credentials. **Subsystem affected** server storage (S3 provider) and heartbeat run completion. **Current behavior** Restart-chaos wave w-20261110T041556-s501 (`--storage s3 --storage-faults ST5`): one 41 s window of 403 `ExpiredToken` failed 4 of 6 tasks. Each run ended `adapter_failed` with "The provided token has expired.", and the automatic retries failed within one second, so the tasks needed a board Retry (R3 `retry_did_not_complete`). **Proposed behavior** A credential rejection is a transient infrastructure condition. Critical-path storage calls wait it out like an outage, and a run that stops on it gets the `storage_unavailable` wait retry. After each rejection the S3 client re-resolves its key with `forceRefresh`, so the next attempt signs with a rotated key. The run error names the credentials. If they never come back, the exhausted-wait message tells the operator to check the storage service and its credentials. **Reason and benefit** Expiring or rotated keys are normal in cloud deployments. They must not fail work. ## What Changed - `storage-unavailable.ts`: `ExpiredToken`, `ExpiredTokenException`, `TokenRefreshRequired`, `InvalidToken`, `InvalidAccessKeyId`, `InvalidClientTokenId` and `SignatureDoesNotMatch` answered by the store with 400/403 count as storage unavailable. `AccessDenied` and a credential chain that cannot load any key (`CredentialsProviderError`) stay permanent. New credential-specific run message; the exhausted message also names credentials. - `s3-credential-refresh.ts` (new): S3 client middleware that calls the client's credential provider with `forceRefresh` after a rejection, once per credential generation (requests that signed with an already-replaced key do not refresh again), then rethrows so the existing storage wait owns the retry. - `s3-provider.ts`: installs the middleware on both S3 clients. - `run-completion.ts`: uses the credential-aware message. ## Verification - `cd server && npx vitest run src/__tests__/storage-s3-credential-rotation.test.ts src/storage/ src/__tests__/storage-outage-wait.test.ts src/__tests__/storage-s3-transient-retry.test.ts src/__tests__/storage-s3-failure-diagnostics.test.ts src/services/heartbeat/run-completion.test.ts` (186 pass). - The new `storage-s3-credential-rotation.test.ts` fails 5 of 6 cases on master (red) and passes with this change (green). It covers a real key rotation through the default credential chain, 403 windows for each code, the message, and `AccessDenied` staying permanent. - `cd server && npx tsc --noEmit` passes. - `s3-credential-refresh.test.ts`: two parallel requests signed with the old key, rejected 50 ms and 400 ms apart, cause one forced lookup (3 before the generation check, 2 after). - Restart-chaos rerun on a build with this fix and #20455: w-20261110T050044-s503 (`--env local --workloads A1,C1 --faults K1,K2 --tasks 8 --storage s3 --storage-faults ST5,ST5R --storage-fault-seconds 30-90`) passed 8/8 tasks (R1–R6 and T) with a 39 s `ExpiredToken` window and an 85 s real key rotation (76 credential 403s). Before the fix, w-20261110T041556-s501 failed 4/6 tasks. ## Risks - Scope: calls with a Readable body (streamed `putObject`, buffered multipart) cannot be resent, so they still fail on the first rejection, as they do for outages. The middleware has already re-resolved the key, so the caller's next attempt (the run-level wait retry for run work) uses the new key. - A key that the store rejects for good (for example a wrong secret) now waits up to the storage wait budget (5 min in the call, then up to 10 run retries) before the board shows the exhausted message, instead of failing at once. - `forceRefresh` re-runs the credential chain after a rejection. Calls are coalesced, so a burst of rejections causes one refresh. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude
|
bf1524c5f8 |
test(restart-chaos): pin each wave's API URL to its own loopback server (#20462)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart-chaos harness starts an isolated Paperclip server for
each wave and runs real agents against it
> - Agents find the server through `PAPERCLIP_API_URL`, which the server
derives at startup
> - On a developer machine with a public URL for the main instance (for
example a tailnet origin), the wave server derived that public URL
instead of its own loopback origin
> - So `process` agents sent the run key for the temporary instance to a
different host, which returned 502
> - This pull request pins each wave server's `PAPERCLIP_API_URL` to its
own loopback origin and removes the Q3 script workaround
> - The benefit is that a wave's agent credentials only go to the wave's
own server
## Linked Issues or Issue Description
**What happened?**
In isolated restart-chaos wave instances, `process` agents received
`PAPERCLIP_API_URL` set to the machine's public tailnet origin. That
origin serves the developer's main instance, not the wave. The agent's
run key for the temporary instance was sent to that host, which returned
502. The Q3 agent script worked around this with a hard-coded loopback
base URL.
**Expected behavior**
A wave's agents call the wave's own server.
**Steps to reproduce**
1. Configure a public URL (for example a tailnet origin) for the
developer's main Paperclip instance on the machine.
2. Run a restart-chaos wave with a `process` agent that reads
`PAPERCLIP_API_URL`.
3. The agent's API calls go to the public origin, not to the wave's
`127.0.0.1` port.
**Paperclip version or commit**
master at
canary/v2026.1110.0-canary.6
|
||
|
DottaandClaude
|
400bb1f0f9 |
test(restart-chaos): one-row db pressure, bound accepted-Retry waits in U1, live L1 results (#20472)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The restart chaos harness grades real agent tasks on an isolated control plane. > - #20421 added the L1 contention workload, `--db-pressure`, and the universal checks U1–U4. > - The first live L1 waves showed two harness gaps and one missed product shape. > - The pressure writers locked several run rows per statement and deadlocked with product transactions. A real stream does not do that. > - U1 counted an accepted Retry that waits on the system as a live owner forever. Ten tasks sat 8 minutes in that state with no run. > - This pull request fixes both gaps, adds the owed direct test for `collectContentionEvidence`, and records the live wave results. ## Linked Issues or Issue Description Refs #20421, #20385. **What happened?** The live L1 waves (`w-20261110T040531-s20385` on the #20385 parent, `w-20261110T044338-s20385` and `w-20261110T050818-s20386` on master) produced harness deadlocks (40P01) from multi-row pressure statements. U1 passed tasks whose accepted Retry waited on `process_identity_missing` for 16 minutes without a run. **Expected behavior** Pressure writers behave like a run-log stream (one row per statement). U1 flags an accepted Retry that waits on the system past the queue bound. **Steps to reproduce** Run `node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/universal-audit.ts <results dirs>` over the saved waves, or the L1 wave command in the README. ## What Changed - `contention.ts`: each pressure statement updates one running run row. The old statement updated every running row at once and deadlocked with product transactions. - `universal.ts` (U1): an accepted Retry that waits on the system (`retryWait`) is an owner only for the queue bound (5 min of server-ready time). While it waits, the Retry button, failed notice and blocker do not count as actionable items, because the person already acted. A scheduled retry is still a timer. - `contention-db.test.ts`: a real PostgreSQL test for `collectContentionEvidence` (receipt count, duplicate producer events, terminal receipt, accounting fields, company scope). - README: live wave results table, updated audit numbers and the U1 rule. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 28 files, 345 tests pass. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: passes. - Audit over 79 saved waves (1,620 tasks, 1,483 passing): U1 17 flags, none on a passing task or an all-pass wave. The longest accepted-Retry wait on a passing task was 278 s. U2 1, U3 30 (3 on passing tasks, warning only), U4 0. - Live waves (codex and claude, `--workloads L1 --faults K1 --concurrency 4 --tasks 4 --db-pressure 4`): - `w-20261110T040531-s20385`, #20385 parent: 117 receipt-write serialization failures (40001) in the server log; 1 of 4 tasks passed. Three runs failed `adapter_failed` after 40001 exhausted in `createRunUsageRecorder`, `reserveRunBudget` and native finalization. - `w-20261110T044338-s20385`, master `8004ad069e` with the harness: 2 serialization failures; 1 of 4 passed. Failures included 40001 in `createRunUsageRecorder` and deadlocks caused by the old multi-row writers (fixed here). - `w-20261110T050818-s20386`, same build with one-row writers: 0 of 4 passed. Every first run failed in `reserveRunBudget` before spawn, and every accepted Retry then waited on `process_identity_missing` without starting a run. - No wave reached the runner's commit stall, so none showed `native_event_replay_conflict`. ## Risks - U1 can now fail a task whose accepted Retry waits more than 8 minutes. No passing task in 79 saved waves came close (longest 278 s). - One-row writers produce fewer statements per second than the multi-row statement did; `contention-db.test.ts` still reproduces the #20385 receipt race with them. - Test-only change; no product code. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, file editing, GitHub CLI). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
2444658c6f |
test(restart-chaos): add ST5 credential rejection and ST5R key rotation storage faults (#20455)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart-chaos harness runs real agent tasks while it injects infrastructure faults > - `--storage s3` already puts a fault proxy between the server and a loopback MinIO (ST1 outage, ST2 latency, ST3 errors) > - Cloud deployments use the default AWS credential chain, and keys can expire or rotate during a run > - The harness had no fault for a rejected or rotated key, so this failure class was untested > - This pull request adds ST5 (a window of 403 credential rejections from the proxy) and ST5R (a real key rotation at MinIO) > - The benefit is that waves can prove runs wait out credential problems and pick up rotated keys ## Linked Issues or Issue Description **What existing behavior does this improve?** The restart-chaos storage faults cover outages, latency and 5xx errors only. **Subsystem affected** tests/restart-chaos (storage chaos). **Current behavior** No wave can reject the server's S3 credentials or rotate the store's key. **Proposed behavior** `--storage-faults ST5` answers every request in a window with 403 `ExpiredToken`, `InvalidAccessKeyId` or `SignatureDoesNotMatch`. `--storage-faults ST5R` restarts MinIO with a new root key (the old key stops working) and gives the server the new key through its credential source only when the window ends. **Reason and benefit** The first local ST5 wave (w-20261110T041556-s501) shows the gap: a 41 s `ExpiredToken` window failed 4 of 6 tasks with `adapter_failed`, and the automatic retries failed within one second. The product fix is a separate PR. ## What Changed - `tests/restart-chaos/storage-credentials.ts` (new): ST5 and ST5R rule planning, a wave-owned `AWS_CONFIG_FILE` profile whose `credential_process` prints the current key, MinIO restart with a new root key on the same loopback port, and a proxy wrapper that holds traffic during the restart and publishes the new key when the window ends. - `storage-proxy.ts`: the `expired_credentials` fault takes an S3 error code; a `credential_rotation` marker passes traffic through. - `storage-chaos.ts`, `types.ts`, `config.ts`, `main.ts`: registry lines for ST5/ST5R and the rotation wiring. - README: ST5/ST5R rows and the credential-source description. ## Verification - `PAPERCLIP_IN_WORKTREE=false npx vitest run --config tests/restart-chaos/vitest.config.ts` (28 files, 328 tests pass). The new `storage-credentials.test.ts` fails before this change (the kinds do not exist) and passes after. It includes a real MinIO test: after a rotation the old key gets 403 `InvalidAccessKeyId`, a client with a cached key fails, and the same client succeeds after it re-resolves its profile. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos` passes. - Live wave: `--env local --workloads A1,C1 --faults K1,K2 --tasks 6 --storage s3 --storage-faults ST5,ST5R --storage-fault-seconds 30-90` ran and injected 62 credential 403s (storage-proxy.json). ## Risks - Test harness only. No product code changes. - ST5R restarts MinIO. Requests are held for up to 5 s during the restart, so a slow restart can also show as a short 502 blip. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Nicky Leach
|
8809f44b5e | Connect Codex and Claude login to agent computers (#20417) | ||
|
DottaandClaude Opus 5.5
|
95cb3c94c4 |
test(restart-chaos): cap store bandwidth during ST2 slow-store windows (#20452)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart-chaos harness tests how runs survive restarts and cloud storage faults > - A Paperclip-sized remote workspace (R1 on Daytona with `--storage s3`) saves about 730 MiB per snapshot as a multipart upload and restores it as one large download > - The ST2 "slow store" fault only adds latency per request. On a healthy loopback store the whole multipart upload takes about 2 s, so a latency fault adds a fixed delay per part and never makes a large transfer slow > - A real degraded store is slow per byte too. We need that to test large saves and restores that take minutes > - This pull request gives each ST2 window a seeded bandwidth cap that all requests in the window share > - The benefit is that ST2 waves now slow large snapshot uploads and downloads by minutes, and the harness can find save and restore timeouts on big workspaces ## Linked Issues or Issue Description Refs #20299, #20313 (storage outage handling on small workspaces), #20442 (saved workspace Files over 512 MiB). **What existing behavior does this improve?** The ST2 slow-store fault in the restart-chaos storage proxy. **Subsystem affected** Test harness (`tests/restart-chaos`). **Current behavior** ST2 adds 1–10 s of latency before each request. The bytes then flow at loopback speed. A 730 MiB snapshot upload or download is still fast. **Proposed behavior** ST2 also caps bandwidth at a seeded 2–16 MiB/s. One rate per direction is shared by every request in the window, so parallel multipart parts share the cap. **Reason and benefit** Large saves and restores now take minutes under ST2. This shows the save and restore timing, the storage request counts, and any timeout on a slow store. ## What Changed - New `tests/restart-chaos/storage-bandwidth.ts`: a shared byte-rate pacer (`ByteRate`), a pacing stream (`paced`), and the seeded ST2 bandwidth draw (2–16 MiB/s). - `storage-proxy.ts`: the `latency` fault takes an optional `bytesPerSecond`. The proxy paces the request body and the response body through one rate per direction for each rule. - `storage-chaos.ts`: ST2 windows set `bytesPerSecond`. Saved receipts keep the rule, so a replay uses the same cap. - README: the ST2 row describes the cap. ## Verification - `PAPERCLIP_IN_WORKTREE=false npx vitest run --config tests/restart-chaos/vitest.config.ts tests/restart-chaos/storage-bandwidth.test.ts tests/restart-chaos/storage-proxy.test.ts tests/restart-chaos/storage-chaos.test.ts`: 14 pass. - Red before: with the proxy and ST2 changes reverted, the proxy cap test and the ST2 draw test fail. Green after. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: pass. ## Risks - Low risk. Test harness only. ST2 now draws one more seeded number, so a seed gives different later windows than before. Replays read the saved receipts and are not affected. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
30d99f4e32 |
test(restart-chaos): add time faults (TM1-TM3): freeze, sleep with aged rows, aged outage (#20458)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The restart chaos harness (`tests/restart-chaos/`) tests if agent tasks survive control-plane restarts. > - Each wave runs for minutes on a fresh instance. Nothing time-bound ever expires during a wave. > - Production failures on 2026-11-09 were about time: an OAuth refresh after more than 1 h made pending approvals invalid (#20340), and a timeout used the wrong window (#20330). > - This pull request adds a fourth fault family, time faults (TM1–TM3). The faults make leases, deadlines and waits expire during a run or across a restart. They do not change product code. > - The benefit is that waves now find failures that occur only when time passes. The first waves found one: a run whose turn completes after a host sleep fails, and the task needs Retry. ## Linked Issues or Issue Description Refs #20340 Refs #20330 **What is the problem?** The chaos harness never moves the clock, so it cannot find failures where a lease, a deadline or a pending wait expires. **What is the solution?** Add three time faults. Each one makes stored deadlines lapse in a way that a real host sees: a process pause, a host sleep, or a long outage. **Why is it needed?** Two days of chaos testing did not find the failures of #20340 and #20330, because no wave lasted long enough for a deadline to pass. ## What Changed - Add `TM1`, `TM2` and `TM3` to `--faults` (`time-faults.ts`, `time-faults-config.ts`): - **TM1 freeze**: send `SIGSTOP` to the server process only, for 95–150 s, then send `SIGCONT`. The runner and Postgres continue. The pause is longer than the controller-instance lease (90 s) and the execution-control deadline (60 s). - **TM2 sleep**: send `SIGSTOP` to the server and all of its descendants. While they are stopped, move every database time back by the age. Then send `SIGCONT`. This is a stand-in for a laptop sleep. - **TM3 aged outage**: send `SIGKILL` to the server tree (as K1). While it is down, move every database time back by the age. Then start a replacement. With `--workloads G1 --triggers approval`, an approval stays pending for a day across a restart. - Add `--time-age-hours` (1–72, default 26). TM faults run only with `--env local`, and not with storms, crash points or K14. - Aging is one transaction. It moves every `timestamp` column, and every time inside a `jsonb` column (ISO strings in their own format, and epoch milliseconds in the wave's window), back by the same interval. Triggers are off for the shift. Copies of a time in JSON (for example the comment `updatedAt` in a continuation receipt) still match their columns. The first wave showed that a shift of the columns only causes false failures. - The identity, state and time columns of the time-bound tables are written before the shift to `<wave>/time-faults/<faultId>.json`. Per-table counts go to the fault receipt (`timeFault`) and the timeline (`time_fault_aged`). - The observer and grader read through `TimeShiftedApi`. It adds the age back to each time that a shift moved, so R4, R5, U1–U4 and the speed budgets use real time. - Make small additions to the registries: `types.ts` (fault kinds, receipt field, option), `config.ts`, `scheduler.ts` (a seeded freeze length only for TM tasks, so the plans of existing seeds do not change), `supervisor.ts` (the freeze and aging hooks), `main.ts` (one line for the API) and the README. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 30 files, 368 tests pass. The new tests are: - `time-faults.test.ts`: option parsing, freeze order, Postgres exclusion, statement generation, and reading times back at their real time. - `time-faults-db.test.ts`: aging runs on real embedded Postgres. The test checks that trigger rewrites are suppressed, that JSON copies match their columns, that text columns do not change, that the evidence file is written, and that an age the wave nearly outlived is refused. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos` passes. - Live waves (local, codex), each run from a separate worktree and one wave per slot: | Wave | Scenario | Result | | --- | --- | --- | | `w-20261110T031929-s301` (A1,C1 × TM1–TM3) | First run, before JSON aging | t01: TM1 missed, because A1 finished first. t02: pass. t03 (C1): its turn completed after TM2 and failed with `native_session_lease_lost`; a follow-up comment woke a new run. After TM3, that run failed with `continuation_user_authorization_missing` and needed Retry. That second failure was a harness artifact: the receipt kept the comment's `updatedAt` in JSON, which the first version did not shift. JSON aging fixes it. | | `w-20261110T041910-s303` (B5 × TM1–TM3) | Freeze, sleep, aged outage | t01 and t02 crossed TM1, then TM2. Both turns completed after the thaw and failed with `native_session_lease_lost`, so both needed Retry (R3). This is a **product failure**; see below. t03 (TM3, 26 h aged outage): **pass**, with no Retry. | | `w-20261110T050125-s302` (G1 × TM3, `--triggers approval`) | Aged wait: an approval pending for 26 h across a restart | **Pass** (2/2). Both approvals were still pending after the restart and the board decided after the fault: t01 rejected, so the task is cancelled; t02 approved, so it is done with exactly one subtask. No Retry; U1–U4 pass. | | `w-20261110T051535-s304` (B5, TM2 with `--time-age-hours 1`, then TM1; one task at a time) | Attribute the failure | t01 (TM2 sleep, only 1 h): its turn completed after the thaw, then failed with `native_session_lease_lost`, so the task needed Retry. This reproduces the product failure. t02 (TM1 alone, 113 s server-only freeze): **pass**. The failure follows the database clock jump, not the pause. | **Product failure found (not fixed in this PR).** A local native run whose host sleeps longer than the 20-minute execution lease loses its completed work. After the wake, the provider finishes the turn and the brief is saved. Then result acceptance (`native-session-executor.ts`, the transaction at about line 12031) requires `native_run_finalizations.lease_expires_at > clock_timestamp()`. The database clock has moved on, so the run fails with `native_session_lease_lost`, and the task shows "Couldn't finish this attempt … Retry". No other boot claimed the lease: the owner, attempt and controller identity are unchanged, and the server's monotonic deadline did not expire. The lease is fenced by wall-clock expiry alone. A safe fix changes lease fencing semantics, which is a heartbeat-boundary decision, so I report it here rather than include a fix in this harness PR. Reproduce with `--workloads B5 --faults TM2` (`--time-age-hours 1` is enough; see `w-20261110T051535-s304`). ## Risks - Harness-only change; no product code changes. Waves that do not use TM faults are unchanged. `TimeShiftedApi` returns responses unchanged when no shift was applied. - `SIGSTOP` goes only to the wave server's own process tree. The tree is found from the server PID, and Postgres is excluded. Every frozen process gets `SIGCONT` in a `finally` block and in a process `exit` hook. - Aging is a model: the database becomes what it would be if everything happened the age earlier. Times outside Postgres (in-memory timers, runner state files), times inside `text` columns, and HMACs over a time are not moved. The README lists these limits. Agent JWT expiry is not covered: native runs mint a token for each tool call. ## Model Used - Anthropic Claude, model ID `claude-opus-5-5`, used through Claude Code with tool use (shell, file edits) and extended thinking. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Devin FoleyandPaperclip
|
549ca46dae |
fix(runner): retain Claude GitHub broker settings through the Rust sidecar (#20463)
## Thinking Path > - Paperclip lets people manage AI agents for work. > - Shared agents use the instructing person's GitHub identity for managed commands. > - Native Claude starts through the Rust daemon and a TypeScript ACPX sidecar. > - The Rust launch filter kept the command launchers but removed their broker transport. > - Shell commands then reported `GitHub capability_missing`, although connector tools worked. > - This pull request retains that transport for managed Claude launches. > - Each command can acquire its current identity through the existing broker checks. ## Linked Issues or Issue Description **What happened?** Native Claude could use GitHub connector tools but could not authenticate `git` or `gh`. The Rust daemon removed the broker URL and session/bridge capabilities before starting the ACPX sidecar. Reconnecting the account did not repair this process environment. **Expected behavior** Managed shell commands use the current authorized GitHub identity. The provider receives broker capabilities, while raw GitHub tokens remain confined to individual commands. **Steps to reproduce** 1. Start native Claude through the Rust Runner with managed GitHub launchers. 2. Run `gh api user` and a private repository ref lookup through Bash. 3. Observe `capability_missing` and no request to the broker. **Paperclip version or commit** Reproduced from source at `2c2186706ef4bb352b662f9d8faebefe47fb52ac`. The repair is based on `a5e01634f69f18dfe10a0bf709df02d9e6a2135b`. **Deployment mode** Native Claude through ACPX. Local acceptance used macOS arm64 and Node 25.6.1. The reported failure also affected a Linux sandbox. Related: #18212 preserved launcher paths. #18850 repaired the TypeScript filter. This change repairs the preceding Rust filter. The duplicate search found no open repair for this boundary. ROADMAP.md already lists managed personal GitHub identities. ## What Changed - Retain the broker URL and both broker capabilities when Claude has a nonempty managed launcher directory. - Keep raw GitHub tokens, unrelated secrets, empty launcher configurations, and sibling ACPX providers outside that exception. - Exercise the regression through the public sidecar initialize protocol in the existing Rust integration suite. Document the restart requirement. The Rust sidecar transport owns this filter. Credential resolution, company access, approval, and run identity policies keep their existing owners. No schema or API changes are needed. ## Verification - Red/green: the subprocess regression failed on the missing `PAPERCLIP_GITHUB_BROKER_URL` before the fix, then passed with all three broker settings and the negative controls. - Passed before moving the regression: 615 Rust core tests, with 3 existing ignored tests. After moving it into the integration suite, all 20 sidecar integration tests and the 9 remaining sidecar unit controls pass. The moved regression was also run against the unfixed source and failed on the missing broker URL. - Passed: 85 TypeScript environment, launcher, sandbox, and configured-environment controls. These include operation identity changes, revocation, recovery, and exclusion of capabilities from persisted environment data. - Passed: Runner Rust typecheck and format check, plus the Runner build through `node scripts/agent-preflight.mjs --skip-rust-debug`. - Live before/after: real `claude-opus-5-5` through the actual Rust daemon and built ACPX sidecar reproduced the failure with the old binary. Neither shell command reached the broker. With the fixed binary, `gh` identified the connected account and `git` read private repository refs. A second turn retained a workspace file and authenticated again. After the controlled broker denied access, the next command failed without reusing the prior token. No GitHub writes were made. - Live release cutover: saved a session with the unfixed daemon, suspended it, and restored it with the repaired daemon on `aa9f5b30614c011ebc3fe4509e11173b766c1fea`. The same session and workspace marker survived. `gh` authenticated after restoration, then failed when the controlled broker revoked access. - Live acceptance used an Anthropic API credential and a loopback broker with an authorized operator GitHub credential. This isolates the repaired transport; it does not qualify the hosted subscription credential or production grant resolver. - The first PR review gate did not recognize the in-source Rust test. The regression now lives in the existing integration test file, and the same test-coverage gate passes locally. - `pnpm verify` returned its documented exit 2 and required broad CI coverage. The final head passed the full CI typecheck, build, and test lanes. - Greptile reviewed `64cca69b76073cc3185b28334465dee48fbadb29` at 5/5 with no actionable findings or open review threads. Its check passed on that head. The same head is mergeable with no conflicts. - Final head `64cca69b76073cc3185b28334465dee48fbadb29` passed [full PR CI](https://github.com/paperclipai/paperclip/actions/runs/42761401396), [Windows Rust](https://github.com/paperclipai/paperclip/actions/runs/42761401117), and [isolated Runner compilation](https://github.com/paperclipai/paperclip/actions/runs/42761400973). Typecheck, build, all test shards, Runner checks, browser checks, and aggregate verification passed. No checks are pending or failed; five conditional jobs were skipped. ## Risks - The Claude sidecar now receives the intended broker capabilities. The existing broker must continue to enforce active-run identity and current connection access. Raw GitHub tokens remain excluded from the provider launch environment. - Previous-release checkpoint and profile formats carry forward unchanged. A running provider must retire and restart normally to inherit the repaired environment. No profile reset or data migration is required. - Hosted recovery requires delivery of the updated Runner binary. This PR does not deploy or retry existing tasks. Live acceptance has not exercised Linux sandbox confinement or a GitHub write. ## Model Used OpenAI `gpt-6.1-sol` through Codex, with xhigh reasoning. Used repository inspection, code editing, and shell execution. The context window size is not exposed. Anthropic `claude-opus-5-5` was exercised as the acceptance-test provider. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
a5e01634f6 |
feat(ui): ask self-hosted users to star Paperclip on GitHub after 5 done tasks (#20453)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Many teams run Paperclip self-hosted, so the project grows when those users tell others about it > - A GitHub star is the cheapest way for a user to do that, but the app never asks > - Asking too early is noise. A user who has finished a few tasks knows whether Paperclip helps them > - This pull request adds a one-time, dismissable modal that asks self-hosted users for a star after their company has 5 done tasks > - The benefit is more visibility for the project, and no repeated prompt for the user ## Linked Issues or Issue Description **Subsystem affected** Board UI (dialogs shown in the main app shell). **Problem or motivation** Self-hosted users have no prompt to star the repository. Users who like Paperclip often do not think of it. A prompt at the right moment helps other teams find the project. **Proposed solution** Show a small modal once the selected company has 5 tasks with status "done". The modal shows the Paperclip mark, the headline "You completed 5 tasks!", a short body, a "Star on GitHub" button and a "Not now" button. It shows only once per browser. Closing it in any way counts as seen. **Alternatives considered** A permanent banner or sidebar link. Both stay on screen after the user decides, so they cost attention every day. A one-time modal costs attention once. **Roadmap alignment** This is a small UI addition. It does not overlap with planned core work in ROADMAP.md. No related public issues or PRs found (searched for "star", "github star"). ## What Changed - Added `GitHubStarPrompt` (`ui/src/components/GitHubStarPrompt.tsx`). It reads the health endpoint and shows only on self-hosted instances (never on Cloud), never during onboarding, and only after 5 done tasks in the selected company. It waits for any other open dialog to close. It stores a "seen" flag in `localStorage` when it opens, so a reload does not show it again. - Mounted the prompt in `BoardDialogs`. - Exported the modal-detection helpers (`MODAL_SELECTOR`, `useModalOpen`) from `AnnouncementWell` so both components use the same rule. - Added `PaperclipMark`, the mark alone from the existing lockup geometry. The lockup and the mark now share one path constant. - Added the `issues.starPromptDone` query key. - Added tests for when the prompt opens and when it stays hidden. ## Verification - `pnpm --filter @paperclipai/ui exec vitest run src/components/GitHubStarPrompt.test.tsx src/components/AnnouncementWell.test.tsx` - `pnpm check:token-gates` (all gates clean) - `pnpm typecheck` - Manual: on a self-hosted dev instance, mark 5 tasks in one company as done. The modal opens. "Star on GitHub" opens github.com/paperclipai/paperclip in a new tab. Reload: the modal does not open again. To see it again, delete the `paperclip.github-star-prompt.seen` key from `localStorage`. ## Risks - Low risk. The change is UI only and additive. - The prompt runs one small issues query (limit 5) while it is still eligible. After it is seen, the query is disabled. - If `localStorage` is not available, the prompt treats itself as already seen and does not show. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (Anthropic), model id `claude-opus-5-5`, run through Claude Code with tool use (shell, file edits). Standard context window; extended thinking at low effort. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: squadbot <squadbot@squadbots-Mac-mini.local> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>canary/v2026.1110.0-canary.5 |
||
|
DottaandClaude
|
2c2186706e |
test(restart-chaos): human-in-the-loop journeys (Q2, Q3), journey triggers and a journey x fault matrix (#20448)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The restart chaos harness (`tests/restart-chaos/`) restarts the control plane while real agents work, and grades each task. > - Almost every harness workload is "the agent does a file task and finishes". Few workloads have a person in the loop. > - On 2026-11-09, production failures came from journeys with a person in the loop: a saved answer that could not reach a dependency-blocked task (#20363), and an app-action approval that a routine OAuth refresh or a one-hour deadline made invalid (#20340). > - No workload exercised these journeys, so no wave could catch them. > - This pull request adds journey workloads with deterministic oracles and journey-specific fault triggers. It also adds a journey × fault matrix and a way to run a wave against an older build. > - The benefit: one command crosses every journey with every fault kind, and the same wave proves red before a fix and green after it. ## Linked Issues or Issue Description Refs #20363, Refs #20340, Refs #20336, Refs #20432 ## What Changed - New workload **Q2** (`journeys.ts`, `journey-driver.ts`): the agent saves a question with its native `request_human_input` tool. The board adds an unfinished `backlog` prerequisite, marks the task `blocked` by it, and answers through the task-thread respond route. The oracle requires that the answer reaches the agent in bounded answer mode while the task stays blocked, that the prerequisite relation stays, that no run has `issue_dependencies_blocked`, and that no stale Retry notice stays at rest. Then the board finishes the prerequisite and the agent finishes the task. - New workload **Q3**: a harness-owned OAuth 2.1 + MCP app on loopback (`journey-oauth-app.ts`, discovery, dynamic client registration, PKCE, rotating refresh tokens, one write tool). The board connects it through the Apps routes and sets the tool to Ask first for a scripted `process` agent. When the review is pending, the harness moves the review's saved timestamps and the token expiry back 61 minutes in the wave's own database, runs Check connection (which refreshes the token through the app), and approves. The oracle requires one review, a refresh between request and decision, an accepted approval, exactly one `tools/call` with the reviewed arguments, an `executed` request, and an agent report that carries a receipt only the provider result contains. - New triggers `asked` and `answered` fire a restart fault 0–3 s after the saved request is first seen and after the board's response. They work for the existing **Q1** saved question (#20432) too. The board answers before an `answered` fault fires and after every other planned fault. - New `--matrix` option: one task per workload × adapter × (a `K0` no-fault baseline, each restart fault at each of the workload's own triggers, each provider message fault). `K0` is a new task fault kind with no fault. - New `--server-from <checkout>` option: every server start and restart of the wave runs another prepared checkout, for example the parent of a fix, with the same harness code. - Q3 setup allows unpriced usage on the wave's company budget policy, because a scripted process run records usage with no price and the default policy pauses the whole company. - README documents the journeys, the triggers, the matrix and the red-before-fix recipe. Unit tests cover the oracles (pass and the pre-fix failure signatures), the driver's ordering, the matrix plan, option validation and the OAuth app. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 295 passed. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: passed. - Live local waves, codex, one wave at a time (shared machine lock), `--concurrency 2`, at most 3 tasks: | Wave | Server build | Command | Result | | --- | --- | --- | --- | | `w-20261110T024308-s106` | this branch (master `5570debc5b` base) | `--workloads Q3 --faults K1 --matrix --seed 106` | 3/3 pass (K0, K1^asked, K1^answered); the app saw one refresh before each decision and one `tools/call` | | `w-20261110T024447-s106` | `--server-from` parent of #20340 (`0e325e8d52`) | same | 3/3 fail as expected: approve returns 409 `action_expired` ("Tool review has expired"), no `tools/call`, task stays `in_review` | | `w-20261110T024644-s107` | this branch | `--workloads Q2 --faults K1 --matrix --seed 107` | K1^asked and K1^answered pass. The K0 cell failed only because the oracle then required a comment; the agent had reported `ANSWER-…` in its bounded turn's run result 14 s after the answer, while blocked. The oracle now accepts either. | | `w-20261110T030341-s107` | `--server-from` parent of #20363 (`eebdfe71b1`) | `--workloads Q2 --faults K1 --matrix --seed 107` | 3/3 fail as expected: `answer_reached_agent_while_blocked`. No run starts between the answer and the finished prerequisite (6 minutes of ready server). | | `w-20261110T033426-s108` | this branch, rebased onto #20432 | `--workloads Q1,Q2 --faults K1 --triggers asked,answered --tasks 4 --seed 108` | Q1^asked and Q2^asked pass. Q1^answered and Q2^answered fail on a product gap (below). | | `w-20261110T020037-s102` | this branch, before the rebase | `--workloads Q1,Q2,Q3 --faults K1 --triggers asked,answered --tasks 3 --seed 102` | Q1 with `asked` and `answered` K1 restarts: both done with the answer. | Waves `…-s107` (pre-fix) and `…-s106` (pre-fix) used the same harness code as the green waves. Only the server build changed. **New product failure (not fixed here).** A K1 restart 0.3–3 s after the board's answer can lose the answer turn's native run: - `w-20261110T033426-s108/t04` (Q1, K1 2.9 s after the answer): run `a40cc8f7` resumes with `resume_dead_runner`, then fails `runner_state_identity_mismatch: authority_indeterminate` (`adapter_failed`). The task shows "Couldn't finish this attempt. Your message is saved." and needs Retry. Retry works. - `w-20261110T033426-s108/t03` (Q2, blocked task, K1 0.27 s after the answer): run `5b5355de` is `process_lost`. Its bounded retry `92a57f50` fails `native_session_recovery_failed` (`session.snapshot` failed). The Retry click starts `0e6aa9d1`, which fails `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate; retirement_refused=journal_commands_unsettled(turn.start.failed)`. The task then waits with `retryWait.reason=process_identity_missing` and stays blocked without the answer. - Suspected path: native runner restart reconciliation of the runner state identity (`resume_dead_runner`, journal command settlement) in `server/src/services/native-runtime/` and `packages/paperclip-runner/`. Host load was 8–14 during the wave. ## Risks - Test harness only. No product code changes. - Q2 depends on the model following its instructions. The oracle accepts the answer in either a comment or the bounded turn's run result, because both were seen. - Q3 shifts persisted timestamps only in the wave's isolated database. It does not shift unrelated tables, so it simulates elapsed time for the review and the token, not for the whole instance. - `K0` and the new triggers add values to shared registries (`types.ts`, `scheduler.ts`). Existing plans keep their meaning. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude <noreply@anthropic.com>canary/v2026.1110.0-canary.4 |
||
|
DottaandClaude Opus 5.5
|
120017e39d |
fix(workspaces): read saved workspace files in place instead of extracting the whole archive (#20442)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - When a cloud task finishes, its workspace is saved as an encrypted
snapshot and its sandbox is released. The board's Files view then reads
that saved copy
> - Saved Files access downloaded, decrypted and extracted the whole
archive for every request, and refused archives over 512 MiB with
`too_large`
> - A finished task in a Paperclip-sized repository saves a snapshot of
about 730 MiB, so its files could not be listed or opened at all
> - The archive format already records each file's size and SHA-256, and
the snapshot records the manifest digest, so single files can be read
and verified without the rest of the archive
> - This pull request reads the saved archive in place, one verified
file at a time
> - The benefit is that users can open any file of a large finished
workspace quickly, with bounded memory and no whole-archive transfer
## Linked Issues or Issue Description
**What happened?**
In restart-chaos waves on Daytona with a real Paperclip clone, the
committed workspace snapshots were 763,944,535 bytes. After the sandbox
was released, every saved Files read returned 422 `too_large` (506 reads
in total), because `createSavedWorkspaceLoader` refuses archives over
`maxArchiveBytes` (512 MiB). The board Files view uses the same loader,
so it fails the same way.
**Expected behavior**
A user can list and open any single file of a saved workspace of
Paperclip-repository size without the server extracting or buffering the
whole archive. Memory and time stay bounded per request, and integrity
is still verified.
**Steps to reproduce**
1. Run a cloud task whose workspace is a large repository, so the final
snapshot exceeds 512 MiB.
2. Let the task finish and its sandbox be released.
3. Open the task's Files view, or call `GET
/api/issues/:id/file-resources/content?workspace=execution&path=…`: it
returns 422 `too_large`.
**Paperclip version or commit**
`master` at
|
||
|
DottaandClaude
|
5ab98b0063 |
test(restart-chaos): add database contention workload and universal checks U1-U4 (#20421)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The restart chaos harness (`tests/restart-chaos/`) runs real agent tasks on an isolated control plane and grades them. > - Two production failures on 2026-11-09 passed the harness. Its tasks made little output and little accounting, so database write contention never happened (#20385). Its grading was per workload, so a task stuck for a reason no scenario targeted was not caught (#20361). > - The harness needs a contention workload and checks that apply to every task. > - This pull request adds the L1 workload, a `--db-pressure` knob, and the universal checks U1–U4. > - The benefit is that the #20385 failure class and unexplained stuck or misleading tasks now fail a wave. ## Linked Issues or Issue Description Refs #20385, #20383, #20361. **What happened?** The harness ran real embedded PostgreSQL, but no task streamed enough run-log output while usage receipts committed. The #20385 receipt race (SQLSTATE 40001, retained outbox, `native_event_replay_conflict`) could not occur. Also, no rule checked every task for an owner or an actionable item, for retry loops, for diagnosable failures, or for stale notices. **Expected behavior** A chaos wave can create run-row contention between run-log writes and accounting writes and check that accounting stays exact. Every task of every wave gets scenario-independent liveness and notice checks. **Steps to reproduce** Run an L1 wave with `--db-pressure 4` against the parent of the #20385 merge (red), then against master (green). Commands are in Verification. ## What Changed - `contention.ts` (new): workload L1 (five 3,000-line shell outputs as five tool calls, then a brief), its accounting oracle, and `RunRowPressure`, the `--db-pressure <writers>` knob. The writers run `update heartbeat_runs set updated_at = updated_at` on the wave company's running runs, in the wave's own isolated database only. - L1 oracle: no `native_event_replay_conflict` or `runner_event_commit_stalled`; each producer event journaled once; a terminal receipt for each succeeded native run; finished runs accounted; per-run cost equals its cost events; the board's spend equals the cost-event sum with no runs pending accounting; and proof that the task produced contention. - `universal.ts` (new): U1 liveness, U2 no silent retry loop, U3 diagnosable failures, U4 no stale notices. `grader.ts` grades them on every task. U1, U2 and U4 fail a task. U3 is a warning. - `invariants.ts`: `readUniversalDbEvidence` reads failed-run error text and recovery-action causes from the wave database at wave end, for U3. - `universal-audit.ts` (new): runs U1–U4 over saved wave evidence to measure false positives and bound headroom. - `contention-db.test.ts` (new): on real PostgreSQL, under the pressure writers, the pre-#20385 repeatable-read receipt write exhausts its retries with 40001 and the current writer commits. - Small registry edits: `types.ts` (L1 id, `dbPressure`, optional `contention` and `universal` fields), `workloads.ts` (L1 entry), `config.ts` (`--db-pressure`, L1 not a default), `main.ts` (start and stop the writers, read database evidence before teardown), `ledger.ts` (universal verdicts and warnings in the ledger and `SUMMARY.md`). - README: "Database contention" and "Universal checks" sections, with the audit table and bound reasons. ## Verification - `contention-db.test.ts` (real embedded PostgreSQL): with `--db-pressure` writers (4), the pre-#20385 repeatable-read receipt write exhausted its retries with 40001 in 8 of 8 attempts; the current READ COMMITTED `persistUsageReceipt` committed 8 of 8. This is the harness-level red/green for #20385. - Live L1 waves (`--env local --adapters codex,claude --workloads L1 --faults K1 --concurrency 4 --tasks 4 --seed 20385 --db-pressure 4`), red on the #20385 parent `bf818280f2` with this harness overlaid and green on this branch: queued behind the shared one-wave-at-a-time lock and not yet run. The first red attempt failed at startup (the fresh checkout lacked a built plugin SDK; no tasks ran). Live results are not claimed here. - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: all harness unit tests pass, including the new `universal.test.ts`, `contention.test.ts` and `contention-db.test.ts`. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: passes. - Universal-check audit over 74 saved waves (1,592 tasks): U1 7 flags (one C1 stall and six R1 runs queued 8 min behind a retry row, the #20361 shape), U2 1, U3 22, U4 0. None of U1, U2 or U4 flags a passing task or an all-pass wave. U3 flags 3 passing tasks (all `adapter_failed`), so it ships as a warning. ## Risks - U1, U2 and U4 can now fail a task that passed R1–R6 and T. The audit found no such case in 70 saved waves. The bounds are environment-configurable. - `--db-pressure` is opt-in and writes no values. It writes only to the wave's own temporary database. - Other agents edit the same harness. This change keeps registry edits small; `grader.ts` and `invariants.ts` changes are additive. - No product code changes. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file editing, GitHub CLI). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
0c7a628fd1 |
test(chaos): read remote R1 files from the execution workspace and record why a read failed (#20440)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness grades Daytona R1 tasks by reading their files through `GET /issues/:id/file-resources/content` > - In replays w-20261110T010031-s9913 and w-20261110T012332-s9950 the tasks recovered (R1, R3, R6 pass), but all 506 content reads returned 422, even after the closing save left the workspace clean (`snapshotId` set, no writer, `location: null`) > - The harness discarded each error, so the waves could not show which 422 the server returned > - This pull request asks for the execution workspace explicitly and records the last error per file in the task record and the timeline > - The benefit is that the next wave names the exact cause instead of a blind `module_present` failure ## Linked Issues or Issue Description Follow-up to #20348. **What happened?** Every file read for a finished Daytona R1 task returned 422, and the response was discarded. **Expected behavior** Failed reads leave their reason in the evidence. **Actual behavior** Only the outcome, a `null` file, was kept. ## What Changed - File reads add `workspace=execution`, so the server reads the issue's execution workspace and not the `auto` choice. - `readRemoteWorkspaceFile` and `captureRemoteWorkspaceFiles` report each failure through a callback. The observer keeps the last message per file in `record.remoteFileErrors` and logs `remote_file_read_failed` once per distinct message. - The remote-workspace file reader stays in `remote-workspace-files.ts`; `observer.ts` is at 993 lines. ## Verification - `npx vitest run --config tests/restart-chaos/vitest.config.ts`: 263 passed. The test server now requires `workspace=execution`, so the existing remote tests were red before, and a new test records the 422 `workspace_unavailable` message. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: clean. ## Risks - If a Daytona task's files only exist in a project workspace rather than its execution workspace, an explicit `execution` selector reports that clearly instead of falling back. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>canary/v2026.1110.0-canary.3 |
||
|
DottaandClaude Opus 5.5
|
d064bd21e4 |
perf(plugins): pick each lifecycle resource's head in one pass (#20406)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Plugins receive resource lifecycle events (agent, project and agent-runtime changes) through a durable inbox > - The inbox lists an event only when it is the first unacknowledged event of its resource, so each resource delivers in order > - The list query checked that rule with a `NOT EXISTS` for every unacknowledged event. Each check walked the resource's whole acknowledged history > - A resource with a long acknowledged history and a backlog behind a deferred head costs history × backlog probes. On the hosted staging instance three copies of this statement ran for more than 23 minutes each, on a database that was at 6–8 of 8 vCPU all day > - This pull request picks the deliverable head of each resource in one pass > - The benefit is a list call that is linear in the company's events, and database CPU that other requests can use ## Linked Issues or Issue Description **What existing behavior does this improve?** `pluginLifecycleInbox(...).list()` in `server/src/services/plugin-lifecycle-inbox.ts`. The delivery worker calls it in a loop and when it checks for idle work. **Subsystem affected** Plugin lifecycle delivery (server service, PostgreSQL query). **Current behavior** Neon "Active queries" for the staging database on 2026-11-09 showed three copies of this statement at 1,381–1,401 s each. In the same view, trivial primary-key reads waited 100–280 ms. Locally, one resource with 2,000 acknowledged events and a 2,000-event backlog takes 3.5 s for one `list()` call. The cost grows with acknowledged history × backlog. **Proposed behavior** Only the head of a resource is deliverable: its unacknowledged creation if it has one, otherwise its lowest unacknowledged event. This is exactly the set that the old `firstForResource` predicate admits. The query now selects these heads with `DISTINCT ON (resource_type, resource_id)` ordered by `(action <> 'create'), id`, and then applies the page cursor, order and limit. **Reason and benefit** The same fixture lists in 8–24 ms instead of 3.5 s. Long-running copies of this statement no longer hold database CPU that every request needs. **Breaking changes** None. `acknowledge()` keeps its per-event eligibility check. ## What Changed - `pluginLifecycleInbox.list()` uses a `lifecycle_heads` CTE with `DISTINCT ON` to pick one head per resource, then filters by the `afterId` cursor and orders by id with the limit. - New test in `server/src/__tests__/resource-lifecycle-events.test.ts`: one resource with 2,000 acknowledged events and a 2,000-event backlog, plus a second resource. It asserts the exact heads in id order and a list time below 1 s (3.5 s before, 8–24 ms after). ## Verification - `cd server && npx vitest run src/__tests__/resource-lifecycle-events.test.ts -t "acknowledged backlog"` — red on `origin/master` (3,546 ms), green with the change (24 ms, 8 ms). - `cd server && npx vitest run src/__tests__/resource-lifecycle-events.test.ts src/__tests__/plugin-lifecycle-delivery.test.ts src/__tests__/agent-runtime-assignments.test.ts src/__tests__/agent-runtime-hooks.test.ts src/__tests__/project-instance-secrets.test.ts` — the existing ordering, paging, per-plugin isolation, backfilled-creation and late-commit tests pass. (One `agent-runtime-hooks` case is timing-sensitive on a loaded machine and also fails on `origin/master` there. It passed 22/22 on rerun.) - `pnpm verify` selection and CI. - After deploy: Neon "Active queries" no longer shows long-running `resource_lifecycle_events … NOT EXISTS` statements. ## Risks - Low risk. The result set is the same: per resource, the unacknowledged creation, otherwise the lowest unacknowledged id. The cursor still applies after the head is chosen, as before, so paging past failed heads does not skip later events. - Each call still reads the company's events of the requested types once. A company with millions of acknowledged events still pays one linear pass per call. A per-plugin acknowledgment watermark would remove that and is out of scope. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, browser, code execution). Extended reasoning on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
0934b834c4 |
perf(workspaces): release retained cohort parent scratch early (#20447)
## Thinking Path > - Paperclip manages agent work and preserves its workspace files. > - Online migration captures local files and publishes remote workspace authority. > - Each capture verifies encrypted readback, restores the archive and compares the current source. > - A cohort kept every completed parent payload until the whole cohort published. > - A larger cohort could retain many complete copies while later members still worked. > - This pull request releases only finished parent payloads before a capture worker starts another member. > - Nested-child proof, final fresh checks and atomic publication keep their existing owners. ## Linked Issues or Issue Description Refs: #20418, #20437, #20441. **What happened?** A cohort retained each member's capture archive, verified archive, comparison archive and restored tree until atomic publication. The capture worker limit did not limit that retained parent data. **Expected behavior** Release completed parent payloads once their consumers finish. Keep files that a later publication check still needs. **Steps to reproduce** Run the added held-peer regression against the previous owner. It completes one parent, holds a later member and finds all four completed parent paths still present. ## What Changed - Release three named parent archives and the restored tree after a successful online cohort capture, before that worker takes another member. - Keep parent scratch roots and all nested-child verification areas. Keep every final fresh child and publication check. - Retain payloads for archived repository intents. Use the existing writable-tree cleanup for read-only restored directories. - Add real filesystem regressions for early release, retained child proof, late child mutation and the final archived-intent path check. Update the source-fence guide. This branch incorporates the final #20441 owner and both distinct saved-relative-path regressions. Its scratch-specific production change is nine added lines and one removed line. Single imports and saved journal/archive formats stay unchanged. ## Verification - Canonical Node24 preflight passed. The Runner build came from the content cache. After the current-master merge, the preflight checked 1,674 inputs and 1,817 outputs and found unchanged build inputs. Plugin SDK prerequisites passed. - Genuine RED on the unchanged #20441 owner: one held-peer case failed at four retained payload paths; the archived-intent control passed. The run took21.14s, with221 unrelated cases intentionally unselected. The first sandbox-restricted setup attempt hit DNS errors and was stopped; the normal network-qualified preflight then passed. - Initial treatment: four selected PostgreSQL/filesystem cases passed in38.61s, with219 intentionally unselected. The held peer observes released parent payloads and retained child areas. A later child mutation still refuses the whole cohort and publishes no authority or activity. - The archived-intent control observes all parent payloads at the actual final saved-relative-path check. The changed path refuses publication and keeps task/repository state intact. This exercises its actual singleton owner; archived-only sources do not enter the online cohort API. - After a normal merge of current master and the final #20441 test, the five affected cases and server TypeScript passed. No full local matrix was repeated. Initial CI failed the inherited recovered-upload assertion because the new retention test leaked its archive and filesystem spies. That case passed alone but failed in the adjacent three-case sequence (two pass, one fail); scoped `finally` cleanup made the same sequence pass three of three in17.15s, with191 unrelated cases intentionally unselected. No upstream assertion or production code changed. Server TypeScript passed again. All required [CI](https://github.com/paperclipai/paperclip/actions/runs/42746586511) passed on final normal-merge head `604c0f67bf7282ecd3f6e5e5da1c9d8637ef70cc` (56 success, four skipped and one neutral; no pending or failing checks). The previously failing server4/12 passed in10m23s. Fresh Greptile is5/5 on that exact head with no open threads. The merge resolves the duplicate #20441 squash insertion; both tested owner and test files are byte-for-byte identical to repair head `b5c8eed82b6a676985e1a8af785e5df2ae3be2ce`. The prior head received [Greptile5/5](https://github.com/paperclipai/paperclip/pull/20447#issuecomment-6757435885); its archived-cohort coverage warning was [answered using the actual public admission boundary](https://github.com/paperclipai/paperclip/pull/20447#discussion_r4520641017). - `git diff --check` passed. No hosted cleanup, migration, provider action or deployment was performed. ```sh node scripts/agent-preflight.mjs --skip-rust-debug pnpm --filter @paperclipai/plugin-sdk ensure-build-deps node --import ./cli/node_modules/tsx/dist/loader.mjs packages/db/src/test-postgres-command.ts pnpm exec vitest run server/src/services/remote-workspace-cohort-migration.test.ts server/src/services/remote-workspace-migration.test.ts -t 'releases completed parent payload|retains archived intent payload|shares the merged child verifier|rechecks the current archived repository intent path at publication' pnpm --filter @paperclipai/server exec tsc --noEmit ``` ## Risks - This reduces retained parent data. It does not establish a total disk bound. Active capture/encryption/readback files and retained child areas still need measured capacity. - Cleanup failure uses the existing worker abort and peer-join path before publication. Original source files, uploaded objects and journals remain intact. - Archived repository intents still need their restored payload at publication. Those payloads stay retained. No capture limit, budget or integrity guard changes. - Saved operations and archive identities carry forward unchanged. Resume still uses the existing full source and authenticated object checks. ## Model Used OpenAI GPT-6 (Codex), with tool use and code execution. The runtime does not expose a more specific model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
9f00f2b83e |
fix(workspaces): recognize committed imports in linked source scope (#20444)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Online workspace migration fences every local Git source that a capture can reach. > - A linked worktree can remain on disk after its workspace is migrated to cloud storage. > - The source inspector lists local workspace rows, so it then treats that linked worktree as unregistered. > - This pull request uses the existing canonical import journal and snapshot proof to recognize its exact original physical root. > - The benefit is that a later linked workspace can migrate without weakening source ownership or stopping agents. ## Linked Issues or Issue Description **What happened?** After one linked Git workspace commits its canonical migration, its local files and Git registration can remain. The workspace row now uses `cloud_sandbox`. A later migration of a related local workspace fails with `online_workspace_source_scope_unproven` because the original linked root is no longer in the local-row list. **Expected behavior** The inspector can recognize the exact original physical root from the same-company successful migration journal and its committed or superseded snapshot. Unknown or changed provenance must still fail. **Steps to reproduce** 1. Create two local Git workspaces with a shared worktree registration. 2. Complete canonical migration for one workspace while retaining its original local files. 3. Inspect or migrate the other workspace. Before this fix, the inspector returns no source scope. **Paperclip version or commit** Reproduced on `f73ff4ace9` with real PostgreSQL, Git, filesystem, and canonical snapshot storage. **Deployment mode** Self-hosted server. This is a core workspace migration bug. **Relevant logs or output** `remote_workspace_migration_blocked:online_workspace_source_scope_unproven` Refs #20418, #20436, #20441. This fix changes source registration only. ## What Changed - Let the existing imported-source owner match optional exact linked physical roots in addition to its existing nested-child paths. - Require each added linked root to have a finished successful same-company journal, an exact committed or superseded snapshot binding, and the same manifest digest. - Include those proven physical roots in the online scope and its source fingerprint checks. - Retain Git root, backlink, pointer, ownership, company, foreign writer, source lock, and publication reinspection guards. - Keep existing nested-child behavior and all persisted schemas unchanged. Previous-release canonical imports remain readable. No runtime authority comes from an arbitrary provider path or remote authority. ## Verification - Genuine baseline RED: both committed and superseded imported sibling cases return a null source scope after a real successful import. - GREEN: 17 focused real-PostgreSQL/FS/Git tests pass. They cover both imported states, the original issue source fence, changed journal/snapshot/manifest/unfinished provenance, a foreign local alias, unregistered links, backlink/pointer corruption, and a listing change during upload. - Command: `pnpm exec vitest run server/src/services/remote-workspace-migration.test.ts -t 'imported linked Git source|foreign local writer|unproven linked|revalidates the linked Git listing|nested Git wrapper.*shared|nested Git wrapper blocked'`. - Server layered typecheck, module boundaries, and `git diff --check` pass. - Actual controlled staging canary: the canonical API committed both targeted workspace snapshots in 310.249 seconds. The owned job closed with exit 0 and all owned processes joined. A separate read-only inventory confirmed that both workspaces left the legacy provider set. This used a reviewed private administrative artifact; the serving app image and normal release path stayed unchanged. - Final head `1759ac6f40a1b35f79587b246de8b36e7a371537`: 56 successful checks, 4 intentional skips, and 1 neutral status. Greptile and Ponytail both report 5/5 with no findings or unresolved threads. - Initial CI had unchanged Runner-support and workspace-recovery timeouts, plus an unrelated heartbeat plugin-environment timing/fixture failure. The same owning inputs passed on the sibling PR. Each failed job then passed one targeted retry on unchanged product bytes. The original failures remain recorded in the CI history. ## Risks - This recognizes a retained original root only from the existing committed import ledger. It does not treat arbitrary cloud workspace paths as local ownership. - Each physical root still passes the existing Git identity and foreign-scope checks. Publication re-inspects the source scope under the existing source locks. - This adds no object reads, provider calls, recurring work, capture algorithm changes, or data exclusion. The existing import query keeps its current 10,240-row limit and runs during explicit migration inspection only. ## Model Used OpenAI GPT-6.1 Codex, with reasoning, repository tools, and code execution. The host does not expose an exact context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>canary/v2026.1110.0-canary.2 |
||
|
DottaandClaude
|
1b52f359dd |
test(restart-chaos): add config fault waves (CF1-CF3) and the CF4 design (#20446)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness (`tests/restart-chaos/`) starts an isolated control plane, runs real agent tasks and restarts the server during them > - Every wave starts from a new instance with valid default settings and one valid provider > - On 2026-11-09 production failed on state that a new instance never has: a setup command that named a renamed script (#20269), and an account default that did not exist (#20361) > - This pull request adds a "config fault" family (CF1–CF3): the harness breaks one saved reference on a task's own agent or project, restarts the server while the task is parked, repairs the setting and grades the result > - The benefit is that the harness now catches "stale saved setting" failures, including a run that retries forever or a task that stays Queued with no explanation ## Linked Issues or Issue Description Refs #20269 Refs #20361 **What existing behavior does this improve?** The restart chaos harness only tests new instances with valid settings. It cannot see failures that come from saved settings that point at a missing object. **Subsystem affected** `tests/restart-chaos/` (test harness only). No product code changes. **Current behavior** No wave breaks a saved setting. The CF2 failure (#20361) and the CF1 failure (#20269) were found in production, not by the harness. **Proposed behavior** `config-wave.ts` runs CF1–CF3. The oracle is not "the task completes". A task passes when all of these are true: - it completes, or it rests on a message that names the broken setting on the task page within a bound; - no run retries unchanged configuration forever; - no task stays Queued with no owner; - the rest survives a K1 or K2 restart; - after repair and the offered action (connection card, Retry, or agent setup Retry), the task completes. ## What Changed - `tests/restart-chaos/config-faults.ts` (new, pure): the catalog of broken references, the seeded plan, `parkState`, the task-level message check and `gradeConfigTask`. CF1 is a missing setup script. CF2 is a missing personal OpenRouter default. CF3 is a seeded walk over a deleted secret, a missing workspace path, a missing base ref, an unknown model, plus CF1 and CF2. `CONFIG_REFERENCES_NOT_BROKEN` lists the settings CF3 cannot break, and why. - `tests/restart-chaos/config-wave.ts` (new): the wave runner. Each task gets its own agent, credential secret and project. The runner breaks the setting through board routes, launches an A1 brief task, polls, fires K1 or K2 while the task is parked, repairs the setting, uses the offered action and grades the task. It reuses the existing supervisor, seed and A1 oracle. Waves are limited to 4 tasks and a concurrency of 2. The instance is deleted at the end. - `tests/restart-chaos/config-faults.test.ts` (new): unit tests for the plan, the arguments, `parkState` and the grader. These include the CF2 bug signature and the CF1 "message only on the run page" signature. - `tests/restart-chaos/README.md`: a new "Config faults" section and a row in the pieces table. - `doc/plans/2026-11-09-chaos-config-and-aged-state-faults.md`: the CF4 design (an instance aged by an older release, built on the existing K14 `--upgrade-from` machinery). CF4 is not built in this pull request. - There are no edits to `main.ts`, `types.ts`, `workloads.ts`, `invariants.ts` or `grader.ts`. Other agents work on those files in parallel. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 24 files, 291 tests pass (before the last unit test was added). The config-fault suite has 13 tests, and all pass. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: passes. - Live waves (local, isolated instance, one wave at a time): - **CF2 red before #20361** (base `8004ad069e`, before #20361 merged): - Waves `w-20261110T000602-s201-cf` and `w-20261110T004818-s202-cf` fail with `CF2/ai_default:bounded_rest+actionable_message+lifecycle_retrying_unchanged_config+queued_without_live_owner`. - The user sees `Queued`, a `resume` user action, and "Execution configuration check failed. Paperclip could not complete this step. It will retry automatically." There is no card. - A debug run with logging added to the lifecycle failure log (not committed) showed that the underlying error is `ai_connection_default_missing`. Verification retried it every 60 s, more than 20 times. - After the harness connects an account that is installed for the agent, the task completes 35 s later on its own. - The CF2 run after #20361, and the CF1 and CF3 waves on current master, are added below as they finish. ## Risks - Low risk. The change is test harness code and docs only. - A wave costs provider tokens only when the repaired task runs (one A1 brief). - The CF2 repair creates a real OpenRouter account in the isolated instance. The server validates the key live. - The oracle reads the task page through the API (interruption, execution status, cards, agent setup status and notices). It does not render the board UI. A message that appears only in the run's own error text is graded `message_only_in_run_detail`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, file edits, subagents). Standard context window. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
0991d9bda1 |
fix(workspaces): protect migration sources and add guarded recovery service (#20443)
## Thinking Path > - Paperclip manages agents and their task workspaces. > - Online migration saves files before it publishes remote workspace authority. > - A saved migration journal can survive the importing process. > - Git cache expiration did not count that journal as a consumer. > - Expiration could remove a source and change its fingerprint while its import was incomplete. > - This PR protects unfinished imports and exposes a guarded internal service transition for an already affected cohort. > - New imports can preserve the current sources while all old recovery data stays retained. ## Linked Issues or Issue Description **What happened?** Git cache expiration could archive and remove a workspace after an import process ended but before its saved cohort finished. The saved source fingerprint then differed from the current archived workspace. Exact cohort resume correctly refused that changed source. Reopening the workspace cannot recreate the old metadata or fingerprint. **Expected behavior** An unfinished migration remains a durable consumer after process loss. If an older server already expired some sources, one canonical transition can fail the exact unpublished cohort without losing its snapshots, objects, source history, or recovery obligations. New attempts use the current verified sources. **Steps to reproduce** 1. Start a cohort import of eight Git worktrees with local changes. 2. Interrupt it after four snapshot uploads. 3. Let an older server expire two idle source worktrees. 4. Resume the saved cohort. It refuses the changed archived sources. Related: #20429 preserves exact cohort identity when independent companion imports change only the scope proof. #20424 preserves prepared recovery state for archived sources. This PR handles completed source expiration, not a scope-only change. Public searches found related expiration work and no matching compensation owner. ## What Changed - Count saved migration markers and unfinished migration journals as Git expiration consumers. Keep failed migration retention holds as consumers too. - Expose `supersedeExpiredRemoteWorkspaceMigrationCohort` as an internal service API for a privileged operator with a real database handle and reviewed company/cohort inputs. It is called directly by the PostgreSQL/Git test and reviewed one-time administrative qualification. It adds no CLI mode or HTTP route. The existing CLI still refuses a damaged cohort; this PR does not claim automatic CLI recovery. - Require an exact unpublished cohort of up to eight members. It requires completed expiration and an unexpired restore recipe for each missing member. The other members must retain their saved source fingerprints. - Recheck current source and recovery state under source, lifecycle, finalizer, and row locks. One transaction fails all old journals, writes audit records, and removes only their exact matched active markers. - Keep old cohort proofs, operations, snapshots, and object references. Genuine failure receipts use the existing failed-save retention hold. This transition does not report migration success. - Reuse the existing bounded source-lock queue when `sourceLockWaitMs` is set. Keep fast refusal by default. Cancellation targets only this waiter, and current checks still run after admission. - Record predecessor IDs on new attempts and reserve a snapshot generation above the existing ledger maximum. New success does not retire the old hold. - Document the owning transition and retention rules. ## Verification - The two new journal-consumer cases failed under the old expiration predicate and passed under the new predicate. - The full new recovery case passed in 21.12 seconds with real PostgreSQL locks, Git repositories, encrypted local object storage, and archive restores. It covers four retained uploads, two real expirations, unchanged old resume refusal, authority change refusal, transaction rollback, a held native finalizer, idempotence, eight new imports, dirty/untracked file restores, and a retention pass 30 days later with no object deletion. - Server TypeScript and module-boundary checks passed. - On `7cec8c915acfad2d08874eaf26abaa852f560219`, the queued-admission regression failed before forwarding the option (24.07 seconds), then passed (22.25 seconds). It verifies a real held project lock, cancellation that preserves the holder and all journals, release into full compensation, retention, and eight restored fresh imports. Current server TypeScript and module/diff checks passed. The only production follow-up changes are the optional queue input and its existing-owner call. - The earlier `737a0df3a71b10fe48d7bbfe7ffd8ae296b4852c` CI run had one Runner support failure: the unchanged credential-free Python/Pi calibration exceeded its outer 25-second subprocess deadline. That job passed 6,926 other tests and skipped four. The timeout is retained as a nonpass. The changed logical head requires fresh CI and review; no test deadline was raised. - The affected cohort, Git expiration, and eligibility suites passed all 148 cases in 228.41 seconds on `aa589f3f447e36d0277c32f8b44cbe5e84a673be`. Merged current master `a9d0946e64`; the only conflict retained both independent added test blocks. The complete merged cohort suite passed all 30 cases in 70.00 seconds on `737a0df3a71b10fe48d7bbfe7ffd8ae296b4852c`, including upstream larger-cohort controls. Final-head server TypeScript, incremental preflight, module-boundary, and diff checks passed. Unchanged expiration/eligibility evidence remains valid. Fresh `809ab704e619812d8f1bf38cf5de9c65893861cc` CI and review are pending. - On `809ab704e619812d8f1bf38cf5de9c65893861cc`, the complete recovery fixture passed in 21.38 seconds after adding one valid archived recipe with no restore point. It refuses with `migration_supersession_expiration_unverified` and keeps all workspaces, journals, and snapshots unchanged. The same case then verifies queue cancellation/release, full compensation, retained history, fresh imports, and restored bytes. Server TypeScript passed. This follow-up changes only the test and service-scope documentation; product bytes remain identical to qualified `7cec8c915acfad2d08874eaf26abaa852f560219`. - Earlier fixture attempts did not pass. They exposed the selection of a held source and an incorrect expected refusal string. The fixture now selects eligible sources and asserts the canonical authority refusal. Production guards were not relaxed. - Live qualification of the exact owning product bytes completed in 230.419 seconds. All eight old operations became failed with auditable supersession lineage. All eight old snapshot rows remained byte-for-byte identical, and their object references remain retained. The existing source-lock queue waited 1.151 seconds for a normal finalizer handoff. The three complete source/recovery passes took about 218 seconds including their transaction, which explains why the earlier 180-second total budget was insufficient for that measured workload. No source proof or publication guard was skipped. Compensation made no provider or object-storage calls. - Earlier live qualifications remain nonpasses: the 180-second attempt did not finish; its instrumented successor completed source checks and refused a busy source lock before mutation. The successful bounded-queue attempt retained those failure records. Fresh archived/online imports are a separate subsequent step and are not claimed complete here. - The local checks do not claim browser acceptance, app deployment, fleet enrollment, or a completed zero-downtime cutover. ## Risks - This owner mutates migration bookkeeping. It preserves all-or-nothing row comparisons and current source/recovery locks. A changed journal, authority, writer, finalizer, expired recipe, or partial transaction refuses the transition. - Historical snapshots and objects remain pinned after new success. This intentionally retains storage. Release requires a separate reviewed action. - An older importer cannot use the successor generation contract. Use the containing canonical owner for recovery. Normal app delivery must also carry the expiration guard; a private maintenance graph does not patch a running server's cleanup worker. - The migration owner crosses 1,000 lines. The bounded compensation transition remains beside its private source inspection, fingerprint, fence, reservation, and publication contracts. Moving it now would require exporting or duplicating those safety owners. No unrelated refactor or new framework is included. ## Model Used OpenAI GPT-6 through Codex, with reasoning, source editing, Git, and local execution tools. The exact serving model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
29a66d679e |
fix(workspaces): verify archived repository intents before publication (#20441)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Workspace migration moves local files into durable, authenticated snapshots. > - Archived Git workspaces can retain an unmaterialized repository request after their local tree is removed. > - Admission cannot prove filesystem absence from the removed tree. > - This pull request checks strict request candidates against the private restored Git tree and the authenticated snapshot readback. > - Publication rechecks current repository rows without changing the request or task history. ## Linked Issues or Issue Description **What happened?** An archived workspace with an authenticated restoration recipe and an unresolved, unmaterialized repository request fails migration with `repository_preparation_unresolved`. Its original directory no longer exists. **Expected behavior** Migration can preserve the request when the actual private restored tree proves its target absent and every parent safe. It must refuse partial materialization, symlinks, or changed request state. **Steps to reproduce** 1. Archive a Git workspace with a valid saved recipe and a pending repository request. 2. Remove its original local workspace directory. 3. Run canonical workspace migration. The previous implementation refuses before it can inspect the restored tree. **Paperclip version or commit** Reproduced on `1ca3e8ea2a759e511ab2d2659339c6c770a7a2ba`. **Deployment mode** Self-hosted server. This is a core workspace migration bug. **Relevant logs or output** `remote_workspace_migration_blocked:repository_preparation_unresolved` Refs #20416, #20424, #20436. These fixes retain their source, recovery, and ownership guards. ## What Changed - Separate strict repository request metadata from filesystem absence proof. - Admit only strict candidates backed by a canonical archived Git recipe to private materialization. - Verify target absence and safe parents before a new upload and after authenticated archive restore. - Lock and recheck current repository rows before authority publication, including recovered uploaded archives. - Preserve previous-release recipes, request rows, task selection, history, provider references, and unresolved recovery obligations. No persisted schema or public option changes. ## Verification - Genuine baseline RED: both pending and needs-repair archived requests fail on unchanged master. - GREEN: 27 focused real-PostgreSQL/FS/Git tests pass. Includes partial/symlink refusal before upload, request mutation during upload, recovered-upload readback, and existing native F/source checks. - Command: `pnpm exec vitest run server/src/services/remote-workspace-migration.test.ts -t 'archived repository intent|unmaterialized repository request|unresolved repository request|pending repository|unselected archived recipe|archived prepared source|competing native F owner'`. - After rebasing onto current master, all six new cases pass on `0b6854789acc067a44e697acebdc04e51b9afe88`. Server layered typecheck, module boundaries, and `git diff --check` pass. A review follow-up adds one pending-path mutation regression (1/1 real-PG pass) without changing product code. Fresh required CI runs on final head `2a30a4b444108488ac8b51d0c46c340ee15227b9`; fresh Greptile is 5/5. Server7 had an unchanged trace-parent assertion failure; that same shard passed on the previous product-identical head and the linked-source sibling PR. The original failure remains recorded. The single targeted server7 retry passed. Final head has 56 successful checks, 4 intentional skips, and 1 neutral review status. ## Risks - Metadata admission is only a candidate check. A candidate still fails unless the private authenticated filesystem proves absence. - Publication keeps repository rows locked while it checks their current paths against the owned restored tree. A real pending-relativePath mutation now tests this final check. Existing workspace/source/native locks remain in place. - Absence is proved only in SAVED data reconstructed from the retained recipe and snapshot. This does not prove that a request never materialized before archival. Existing Git restore points can exclude ignored `.paperclip-repositories` paths outside recorded checkouts. This PR does not change that preexisting coverage or recover bytes that were never saved. - The original recipe, restore point and checkout references remain unchanged. No files are recovered into or removed from the original directory. No request is marked ready or restored. Partial target files present in reconstructed saved data remain blocked and retain their state. ## Model Used OpenAI GPT-6.1 Codex, with reasoning, repository tools, and code execution. The host does not expose an exact context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
b6f4008f9b |
docs: explain individual agent interaction permissions (#20422)
Document all eleven interaction controls, sparse defaults, CLI administration, ownership rules, pending-delivery behavior, and access boundaries. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
5bad074baa |
feat: add individual agent interaction permissions (#20411)
Add eleven sparse incoming and outgoing restrictions, human CLI/API administration, shared mutation enforcement, and pending-delivery checks. Preserve existing defaults, agent settings, and existing authorization boundaries. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude
|
4158799d1b |
test(chaos): add provider message faults (PV1-PV3) and a saved-question scenario (#20432)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The restart chaos harness (`tests/restart-chaos`) tests that agent work survives faults. > - The harness has two fault families: control-plane restarts (K*) and storage faults (ST*). > - On 2026-11-09, two production failures came from a third family. The provider was alive, but a protocol message was late or missing (#20330, #20336). > - The harness could not produce that family, so it did not catch these failures. > - This pull request adds provider message faults (PV1 delay, PV2 drop, PV3 duplicate) on the real path between the runner and the Codex app-server. It also adds a saved-question scenario (Q1) that uses the faults. > - The benefit: the harness can now find this class of failure, and it fails against both pre-fix commits. ## Linked Issues or Issue Description Refs #20330 Refs #20336 **What happened?** A Codex interruption terminal that arrived after 18 s failed a governed wait (#20330). A dropped terminal after a saved handoff failed shutdown settlement with "Couldn't finish this attempt" (#20336). The chaos harness could not inject either fault. **Expected behavior** The harness can delay, drop or duplicate a chosen provider protocol message for one task. A scenario fails on the pre-fix product code and passes on master. **Steps to reproduce** Run the wave commands in Verification. Run them on master and in a worktree at the pre-fix commit with `tests/restart-chaos` copied from this branch. ## What Changed - `provider-faults.ts` and `provider-shim/`: the PV1–PV3 fault family. - A resolve hook goes into the wave's server through `NODE_OPTIONS`. It answers only the `@openai/codex/package.json` lookup that product code uses to find the local Codex. It returns a stand-in package in the instance temp root, and then removes itself from `NODE_OPTIONS`. - The stand-in `codex` bin is a stdio shim. It starts the real Codex with the same arguments and relays JSON lines unchanged. It changes only the message that the task's rule names: the `turn/completed` after a `turn/interrupt` (`terminal`) or the response to `turn/interrupt` (`interrupt_ack`). Each rule applies once per task. The task nonce binds the rule to the task. - Every decision goes to `<wave>/provider-faults.jsonl`. A PV task fails `provider_fault_applied` when its fault did not happen. - There is no product change. Thus the same harness runs against older product commits. - `question-handoff.ts`: Q1 workload. The agent saves one question with `request_human_input`. After the asking run settles, the board answers through the respond route. The oracle checks that the question was asked once, the asking run succeeded, no run failed, nothing ran before the answer, and the brief has the answer. - PV faults run only with `--env local --adapters codex`, only with workloads that interrupt a turn (Q1, S1, G1), and not in the same wave as restart faults. A PV1 delay passes only when the held message was released. - Registry edits: `--faults` accepts PV1–PV3. New options are `--provider-fault-message` and `--provider-fault-delays` (a sweep, cycled across tasks). The edits are in `types.ts`, `config.ts`, `scheduler.ts`, `workloads.ts`, `observer.ts` (Q1 board step) and `main.ts` (install the seam and add the check). - README section with commands, the seam, and what is not covered. ## Verification - Unit tests: `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`. 23 files and 279 tests pass on the final head. The new tests drive the real shim against a stand-in provider for each kind (delay, drop, duplicate, interrupt ack, once per task, held message discarded on exit, a multibyte character split across writes). They also run the resolve hook in a child Node process. - Typecheck: `node scripts/typecheck-test-projects.mjs tests/restart-chaos` passes. - Live waves (real Runner, real Codex 0.160.0, gpt-6.1-sol, macOS arm64): Pre-fix runs use a worktree at the merge parent with `tests/restart-chaos` copied from this branch. Common arguments: `--env local --adapters codex --workloads Q1`. | Scenario | Product code | Wave | Result | | --- | --- | --- | --- | | 1. PV1, terminal 18 s late (`--faults PV1 --provider-fault-delays 18 --tasks 1 --seed 101`) | master (`8004ad069e`) | `w-20261110T000333-s101` | pass: terminal held 18 s and released, asking run `succeeded`, one question, brief has the answer | | 1. same | `34c1e6469c` (parent of #20330 merge) | `w-20261110T014734-s101` | **fail**: Codex stopped at 14.9 s with the terminal still held, asking run `failed native_provider_terminal_failed`, banner "Couldn’t finish this attempt. Your message is saved.", Retry needed (R3) | | 1. sweep 5/14/25/29 s (`--faults PV1 --provider-fault-delays 5,14,25,29 --concurrency 2 --tasks 4 --seed 102`) | master | `w-20261110T015426-s102` | 4/4 pass | | 1. same sweep | `34c1e6469c` | `w-20261110T015708-s102` | 5 s and 14 s pass; **25 s and 29 s fail** (`native_provider_terminal_failed`, Retry needed): the old 15 s drain | | 2. PV2, terminal dropped (`--faults PV2 --tasks 1 --seed 103`) | master | `w-20261110T015239-s103` | pass: terminal dropped, asking run `succeeded` through verified shutdown, one question, nothing ran before the answer | | 2. same | `a67b936885` (parent of #20336 merge) | `w-20261110T015010-s103` | **fail**: asking run `failed native_provider_terminal_failed`, banner "Couldn’t finish this attempt", Retry needed (R3) | The first master wave exited 1 for one cold `agent_boot` speed sample (8.6 s against a 7 s p95 budget), before any fault, while a preflight build ran in parallel. Its task passed. No new product failure was found on master. ## Risks - Test-only. No product code changes. - The shim adds one Node process in front of Codex for PV waves only. Other waves do not change. - The nonce binds a rule to a Codex process. If one Codex process serves two tasks, the second task's traffic can bind it too. Q1 tasks use separate sessions. - Claude is not covered. Claude runs through the ACPX sidecar against a hash-verified provider pack, and its ACP protocol has no interrupt acknowledgement. A Claude shim needs a different seam. ## Model Used Anthropic Claude Opus 5.5 (`claude-opus-5-5`), through Claude Code, with tool use, code edits and local execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
490d97bf1e |
test(perf): guard Mine against waiting on the company-wide workspace list (#20419)
> Follows #20388 (merged as |
||
|
DottaandClaude Opus 5.5
|
9a4a5066fa |
perf(inbox): stop optional reads from holding Mine rows hidden (#20407)
> Follows #20388, which merged to `master` as `50df9a19b9`. This branch now includes `origin/master`, so its diff carries only this PR's change: `ui/src/pages/Inbox.tsx`, `ui/src/pages/Inbox.test.tsx`, `ui/src/api/approvals.ts` and `ui/src/api/access.ts`. ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Mine inbox is the first page a board user opens to see the tasks that need them > - The page hides task rows until every inbox read finishes, including reads that do not change the rows > - A slow user directory, approvals or join-request read therefore delays every task row, and a stalled read keeps the skeleton up > - Loading is also a layout contract: content that arrives late must not push visible rows down or reset keyboard selection > - This pull request removes the user-directory gate where names only fill row text, and bounds the approvals and join-request gates so a failure or stall cannot hold rows hidden > - The benefit is that a slow optional read no longer hides task rows, and the rows still do not move when late data arrives ## Linked Issues or Issue Description Refs #20388 **What existing behavior does this improve?** The Mine inbox reveal gate waits for every source, including the user directory, approvals and join requests. **Subsystem affected** Board UI inbox (`ui/src/pages/Inbox.tsx`). **Current behavior** If the user directory is slow, every task row waits for it, although user names only fill text in the rows. If approvals or join requests stall, the skeleton stays up until the request ends. **Proposed behavior** Rows reveal on session, experimental settings, `inbox/mine`, dismissals, agents and projects. The user directory fills names in place. Approvals and join requests can delay rows for one attempt of at most 10 s. **Reason and benefit** A slow optional read no longer hides the task list. A failed read shows the existing "Some inbox items couldn’t load" banner with a retry instead of a skeleton. **Breaking changes** None. ## What Changed - User directory: it no longer gates task rows. It still gates rows when they are grouped by assignee, because user names name and order those groups. - Approvals and join requests: they still gate rows (see "Kept gates" below). The gate now covers only the first attempt (`isLoading && failureCount === 0`). That attempt uses a 10 s request timeout (`INBOX_OPTIONAL_READ_TIMEOUT_MS`). After a failure or timeout, rows reveal and the query retries in the background. - Timeout notice: a timed-out approvals or join-request read is a transient error, which the inbox hides. Once the app's retries end, the inbox now shows the "Some inbox items couldn’t load" notice and "Retry missing items" button for a `TIMEOUT` on those reads. Other transient errors stay hidden as before. - `approvalsApi.list`, `accessApi.listJoinRequests` and `accessApi.listUserDirectory` accept optional `RequestOptions` (signal and timeout). Other callers do not change. - Tests in `ui/src/pages/Inbox.test.tsx` ("optional reads and the reveal gate"): - rows reveal while the user directory is pending, and keep their order, their DOM nodes and the keyboard selection when it arrives; - assignee grouping still waits for the user directory; - a delayed approvals read holds rows (the kept layout guarantee); - a failed approvals or join-request read reveals rows with the retry banner and no skeleton; - after one failed attempt, rows reveal while the approvals read retries; - a stalled approvals read times out and reveals rows; - approvals or join requests that keep timing out under the app retry policy (`shouldRetryRequest`) show the missing-items notice and retry button. ### Kept gates Approvals and join requests stay in the gate. On Mine they sort in among the task rows by time (`getInboxWorkItems`). A pending approval is usually recent, so it sorts near the top. If it arrived after the rows were visible, it would push every task row down by one row. I could not reserve space for it, because the count is unknown until the read returns, and an empty reserved slot would collapse and shift the rows the other way. So I kept these two gates, and I bounded them so they cannot hold rows hidden on failure or stall. Keyboard selection already follows the selected row by key, so late data does not reset it. The user-directory test checks this. ## Verification Focused checks on this revision: - `cd ui && npx vitest run src/pages/Inbox.test.tsx src/api/access.test.ts`: all passed (91 Inbox tests). - Red/green: with the #20388 versions of `Inbox.tsx`, `approvals.ts` and `access.ts`, three new tests fail (user directory pending, retry after first failure, stalled approvals). They pass with this change. The other four pin guarantees that already held. - Review fixes: the timeout-notice tests fail without `eb5d55da86` and pass with it. `cd ui && npx vitest run src/pages/Inbox.test.tsx`: 94 passed. - After merging `origin/master` (`54d73aef19`): `cd ui && npx vitest run src/pages/Inbox.test.tsx src/api/execution-workspaces.test.ts`: 101 passed. `pnpm --filter @paperclipai/ui typecheck`: passed. - CI note: on head `e438e16c80`, `ci / e2e shard (3/8)` failed in `planning-mode-visual-verification.spec.ts` (the first task's `task-chat-composer-mode` did not appear within 30 s). That journey runs onboarding into the first task chat. It does not open the inbox or use the changed reads, so I treat it as unrelated. The other 33 tests in the shard passed. The new head reruns it. `pnpm check:token-gates`: clean. - `pnpm verify --base perf/inbox-referenced-workspaces` selected 2,219 Vitest files because `ui/src/api/access.ts` is imported widely. I did not run that broad set locally (the shared host was under heavy load). Required CI supplies that coverage. Performance (same local fixture as #20388: authenticated mode, non-admin member, 8,750 execution workspaces, 38,750 tasks, production UI bundle, headless Chromium, new context per run). The host load average was about 30–48 from other work, so I used alternating pairs. | Scenario | #20388 | This PR | Median paired saving | | --- | --- | --- | --- | | No injected delay, 6 pairs | median 870 ms (747–1,922) | median 1,046 ms (795–2,002) | mixed signs, no measurable change | | User directory delayed 3 s, 5 pairs | median 3,980 ms (3,852–6,048) | median 1,096 ms (698–1,263) | 2,814 ms | | Approvals stalled 20 s, 1 run each | 20,829 ms | 10,737 ms | — | On this fixture the optional reads finish before `inbox/mine`, so this PR does not change the normal cold load. It helps when an optional read is slower than `inbox/mine`, for example under server load. ## Risks - Low risk. The change only removes or bounds waits for reads that do not decide row order. - Assignee names can appear a moment after the rows on a slow user directory. The rows keep their height and position. - If approvals or join requests time out on every retry, the notice and retry button appear. If they time out and later succeed, they insert into the list after the rows are visible. This happens only in that failure path. - The 10 s timeout applies to these inbox reads only. Other pages that read the same data do not change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edit, headless browser). Extended reasoning on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
99a6daf5bc |
perf(auth): resolve a Cloud tenant actor in one database round trip (#20390)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - On Paperclip Cloud, each request resolves its board actor from trusted Cloud headers in `actorMiddleware` > - This runs for every request after the liveness probe, also for static files such as `favicon.ico` and `index.html` > - The resolution ran three database statements one after another: the stale `instance_admin` purge, the membership read and the owner-elevation read > - On a loaded database, each statement waits. Three sequential waits make every request slow, also a request that needs no data > - This pull request keeps the purge and the tenant sync in order and runs the two reads at the same time as them > - The benefit is one database wait per request instead of three ## Linked Issues or Issue Description **What existing behavior does this improve?** Cloud tenant actor resolution in `server/src/middleware/auth.ts` (`resolveCloudTenantActorOnce`). **Subsystem affected** Server authentication middleware (Cloud trusted-header path). **Current behavior** Measured on the hosted staging instance on 2026-11-09 from a signed-in browser, 45 paired samples in 40 s: - `/api/liveness` (mounted before actor resolution): client median 92 ms, p90 255 ms, server time ≤ 1 ms. - `/favicon.ico` with a session (after actor resolution): client median 413 ms, p90 1,751 ms, max 4.6 s. - `/api/cli-auth/me`: server median 346 ms, p90 1,528 ms. The same Node process answers liveness quickly, so the event loop is not blocked. The database is busy (6–8 of 8 vCPU for the whole day), and trivial primary-key reads waited 100–280 ms each. Three sequential reads explain the ~320 ms median gap between liveness and favicon. **Proposed behavior** The purge and the debounced tenant sync stay ordered. The membership read and the owner-elevation read run at the same time as them. The sync writes only the primary company's membership, and the membership read excludes that company, so the result does not change. **Reason and benefit** A synced request costs one database wait instead of three. With 150 ms of injected database latency, a synced resolution goes from 914 ms to 310 ms (3 runs each, same machine). **Breaking changes** None. ## What Changed - `resolveCloudTenantActorOnce` starts the purge-and-sync, the additional-membership read and `resolveOwnerInstanceAdmin` together and awaits them with `Promise.all`. A membership read failure still degrades to the primary company. The owner-elevation read still fails closed. - New regression test `server/src/__tests__/cloud-tenant-actor-round-trips.test.ts`. It connects through a TCP proxy that delays every client write by 150 ms and asserts that a synced resolution finishes in less than 4 × the delay. Before the change it takes about 6 × (914 ms); after the change about 2 × (310 ms). - `auth-session-route.test.ts`: the fake database binds each `returning()` to its own insert, and the test matches inserts by content instead of position, because the instance-settings insert can now happen in parallel with the tenant sync. ## Verification - `cd server && npx vitest run src/__tests__/cloud-tenant-actor-round-trips.test.ts` — red on `origin/master` (914 ms, 913 ms, 916 ms), green with the change (310 ms). - `cd server && npx vitest run src/__tests__/cloud-tenant-company-provisioning.test.ts src/__tests__/auth-session-route.test.ts src/__tests__/liveness.test.ts src/__tests__/cloud-warm-standby.test.ts src/middleware/cloud-tenant-actor.test.ts` - `pnpm verify` (selected tests), and CI. - After deploy: compare `/favicon.ico` and `/api/cli-auth/me` client time with `/api/liveness` on staging. Expect the median gap to drop from about 3 database waits to about 1. ## Risks - Low risk. Authorization does not change: the same statements run and the request waits for all of them before it continues. - Each request now uses up to three pooled connections at the same time instead of one. The staging pool limit is 500, so this is not a concern there. Smaller pools reach their limit sooner under a burst. - The diff re-indents the purge-and-sync block. The statements in it do not change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, browser, code execution). Extended reasoning on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Devin FoleyandClaude Opus 5.5
|
a9d0946e64 |
fix(ui): keep the persisted cache's final state when it cancels reads (#20431)
The persisted read cache cancels in-flight reads with revert: false when it clears account data or evicts a denied task, then writes their final state itself. A non-silent cancel settled later as a query error, which overwrote that state with CancelledError and reached QueryCache.onError. Cancel these reads silently so the written state stays. Refs #20430 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Dotta
|
f73ff4ace9 |
fix(native): reprove stale filesystem preservation receipts (#20438)
## Thinking Path > - Paperclip manages agent work and preserves saved native sessions across releases. > - Filesystem independence receipts bind database facts to preserved controller and skill evidence. > - A workspace migration can add run metadata without changing those preserved bytes. > - That change makes the saved facts digest stale. Full native preservation then refuses the existing receipt, even after fresh source checks. > - WEB preservation already supports a full current proof and guarded receipt refresh. > - This pull request applies that pattern to full native receipts, reuses the authenticated archive, and keeps the existing ownership and destruction checks. > - The benefit is a current receipt without a new archive or provider replay. ## Linked Issues or Issue Description Refs: #20322. **What happened?** Adding the canonical `legacyWorkspaceRecoveryMigration` marker to a settled native run makes its saved receipt stale. The full native preservation owner checks current evidence but then refuses the old facts digest. **Expected behavior** The owner can refresh the proof metadata after it proves all current facts and preserved bindings again. Changed evidence or authority must leave the prior receipt intact. **Steps to reproduce** Use the added PostgreSQL regression. It saves a valid receipt for a succeeded source with 453 checkpoint parts, adds the producer-shaped migration marker, and calls preservation again. The unchanged owner fails with `native_harness_filesystem_independence_unverified`. ## What Changed - Reuse the prior controller archive through the existing authenticated readback path. - Require the old receipt to match every current immutable binding before a facts refresh. Reconstruct its exact digest from unchanged bindings and the prior proof fields. - Recheck current database facts, assets, controller and local files under the existing finalizer, selection, task and run fences. - Publish with an exact prior-receipt comparison. Keep a valid concurrent winner and refuse changed receipts or new owners. - Audit the prior receipt digest and four proof fields. Add focused source, integrity, race and DB-only validation tests, and update the database guide. ## Verification - Local Node24 canonical preflight and plugin SDK prerequisites passed. The Runner build came from the content cache. After a normal merge of fresh master, preflight confirmed unchanged build inputs. - Genuine RED on `3040448da5`: the saved receipt is valid before the migration marker and stale afterward. Reproof fails at the old receipt guard (one failed, 102 intentionally not selected). Both retained RED attempts reached that same guard. - GREEN on final local merge `00f4b5d678083106afc88fb2f7e0588961b46d6d`: 19 PostgreSQL controls passed, 91 intentionally not selected (17.75s). The wrapper completed cleanup. - The positive preserves the same archive, controller bytes, assets, local proof and all 453 checkpoint parts. It performs authenticated object reads and zero archive PUTs. Runs, results, finalizers and leases remain unchanged except for the receipt. The audit reconstructs the exact prior receipt. - DB-only validation passes with the original home, state directory and volume absent, while storage reads throw. - Refusal and race controls cover changed asset bindings, raw controller bytes, missing destruction evidence, unknown receipt fields, a valid concurrent winner, changed receipts and a late active owner. Existing WEB refresh, physical finalization-lock and saved-binding controls also pass. - Server TypeScript passed, including the adjacent integration test. Fresh Greptile is 5/5 on exact head `00f4b5d678083106afc88fb2f7e0588961b46d6d`, with no review threads ([review](https://github.com/paperclipai/paperclip/pull/20438#issuecomment-6756548830)). Final-head CI is terminal: 56 success, 4 skipped and 1 neutral. The aggregate `ci / verify` passed ([run](https://github.com/paperclipai/paperclip/actions/runs/42737704110)). - The first Runner-support job had one unrelated process-inspection nonpass: `server-stop.test.ts` expected a successful wrapper close, but its bounded process inspector returned no table. Its cause remains unknown. The unchanged test, process helpers, workflow and dependency inputs match the passed master control. One authorized same-head job retry passed all 328 files / 6,927 tests (4 skipped); the exact previously failed case passed in 665ms ([retry](https://github.com/paperclipai/paperclip/actions/runs/42737704110/job/129576420893)). No timeout, assertion or source change was made for the retry. ```sh node scripts/agent-preflight.mjs --skip-rust-debug pnpm --filter @paperclipai/plugin-sdk ensure-build-deps node --import ./cli/node_modules/tsx/dist/loader.mjs packages/db/src/test-postgres-command.ts pnpm exec vitest run server/src/services/native-runtime/native-harness-filesystem-independence.integration.test.ts -t "reproves stale native facts|retains stale native proof|guards stale native reproof|invalidates a published receipt after changed|actual physical-finalization lock|refreshes stale WEB facts|refuses a WEB facts refresh|retains or refuses the existing receipt when an earlier capture finishes late" pnpm --filter @paperclipai/server exec tsc --noEmit ``` ## Risks - Old receipt schemas and immutable bindings remain required. Full fresh proof carries compatible saved receipts forward; unknown fields or changed evidence refuse refresh and preserve the old receipt. - Full native preservation still requires canonical destroyed-provider receipts. The change retains the existing source capability, cancellation, finalizer and publication checks. - Only proof metadata changes. No provider operation, archive replacement, hosted recovery or deployment was performed for this PR. Live qualification remains with the operator after delivery. ## Model Used OpenAI GPT-6 (Codex), with tool use and code execution. The runtime does not expose a more specific model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting mergecanary/v2026.1110.0-canary.1 |
||
|
DottaandPaperclip
|
07f6e8ff75 |
perf(workspaces): group same-root cohorts with bounded workers (#20437)
## Thinking Path > - Paperclip preserves agent work when its control-plane server is replaced. > - Online migration imports local workspaces into authenticated snapshots. > - Several workspace records can point to one physical directory. > - Shared child verification avoids repeated work inside one held cohort fence. > - The eight-member limit splits 30 identical-root aliases into four cohorts and repeats the child passes. > - This pull request admits up to 32 members only for one canonical physical root, while keeping each work stage bounded to eight peers. > - The benefit is fewer repeated child checks while capture and publication guards stay in place. ## Linked Issues or Issue Description Refs: #19873, #19882, #20418, #20424, #20429. Related PR search found shared verification and restart fixes, but no larger same-root cohort owner. The shared verifier and archived-source changes are merged into this branch from current master. **Current behavior** Thirty fresh aliases of one physical root require cohorts of 8, 8, 8 and 6. Each cohort performs an initial and final local child-validation pass. The same children are checked eight times in total. **Proposed behavior** Admit one cohort with nine to 32 members only when every canonical resolved physical root is exactly equal. Keep the existing complete source-scope equality and admission guards. Keep mixed-leaf cohorts limited to eight members. Bound source reads, reservations, recovery checks, final readers, replay and cleanup to eight joined peers. Capture concurrency stays at most eight. **Reason and benefit** For N distinct canonical covered children, the 30-alias case changes 8N local child checks to 2N, a 75% reduction in repeated child validation. If N is 68, that is 544 checks versus 136. Thirty parent captures, uploads and restores remain required. No total wall-clock improvement is claimed. **Breaking changes** Saved cohort fields and exact membership remain unchanged. Older tools that enforce the eight-member bound refuse larger cohorts. Use the newer tool to resume them; never split or regroup their saved membership. ## What Changed - Raise the canonical service, saved-cohort and maintenance-request member bounds to 32. Keep the capture-worker bound at eight and single-workspace modes at one. - Require one exact canonical physical root for more than eight members. Keep the existing full source-scope comparison, leaf rule and P/I/L/F guards. - Use one private stage mapper with batches of at most eight. Join every started peer before an error can leave the stage. Stop future batches after a failure or cancellation. - Preserve the existing cancellation and pending-snapshot ordering for stages of at most eight members. Check cancellation around extended batches. - Keep all saved cohort, member, operation and snapshot identities, proof refresh and atomic publication checks. - Bound cleanup without skipping later scratch cleanup after an earlier cleanup error. - Extend the existing PostgreSQL fixture with ten real aliases, partial-upload resume, cold restoration, replay and mixed-root refusal. Extend the request boundary test. - Document old-tool refusal, work counts and the retained-scratch capacity bound. ## Verification - Canonical Node24 worktree preflight and Plugin SDK prerequisites passed. After merging current master, preflight passed again with unchanged cached build inputs. - Genuine RED on the old bound: both real ten-alias import/resume cases and the 32-member request reject the eight-member limit (3 failed; 47 not selected; 18.23 seconds). - First focused GREEN: 4 passed in 24.20 seconds. The fresh ten-alias case performs exactly four local child captures for two children: one initial and one final pass. Actual reader, transaction and upload maxima are eight. All ten archives cold-restore after the original source is parked; replay retains the same cohort. - The partial-upload case keeps all ten saved operation IDs and the existing snapshot IDs. More than eight members across different canonical leaf roots refuse before any journal or storage work. The request parser accepts 32 members and eight workers, and refuses 33 members or nine workers. - The first expanded corpus had 55 passes and one real compatibility failure: a new eager cancellation check skipped the old pending snapshot boundary. The mapper now retains existing cancellation ordering for at most eight peers. That failure remains recorded as a nonpass. - Final sources include current master at `18697b8d21`, with the verifier, archived-source and UI repairs. The selected four-file corpus passes 47 cases in 78.88 seconds, with 191 intentionally not selected. The full maintenance suite separately passes 23 cases in 10.21 seconds. Nine cases overlap; the final union is 61 distinct passing checks. - Final coverage includes all 27 cohort cases, all 23 maintenance cases, all six actual old-format compatibility cases and five archived prepared-source controls. Existing source mutation, finalizer contention, actual fence loss, cancellation peer joins, journal compare-and-swap and outer-commit rollback controls pass. - Final server TypeScript, module boundaries and whitespace checks pass. Only the resource-risk documentation changed after that test run; runtime and fixture bytes stayed unchanged. - Commands: `pnpm exec vitest run server/src/services/remote-workspace-cohort-migration.test.ts server/src/maintenance/online-workspace-migration.test.ts server/src/__tests__/online-workspace-migration-compatibility.test.ts`; archived controls use the existing `remote-workspace-migration.test.ts` cases named `unselected archived recipe` and `archived prepared source`; `pnpm --filter @paperclipai/server exec tsc --noEmit`; `node scripts/check-module-boundaries.mjs`. - Initial source review was 4/5 and requested a control for the extended reader-batch stop boundary. The test-only follow-up `ca80ae9cc8791a72dab632090488093c5d0916f3` adds actual ten-member failure and cancellation cases. They hold a first-batch peer, verify that the source fence stays held, and prove members nine and ten never start preflight. Both pass in 10.87 seconds with no new journal, snapshot or upload effects. Runtime source hashes are unchanged; the local union is now 63 distinct checks. - The first stop-test attempt observed the two controlled readers before all SQL-backed peers had entered the inspection callback. Its assertion was too early. The corrected test checks the held first-batch readers at that point, then requires all eight to join before completion. Both cases pass; the earlier attempt remains a nonpass. - Final head `ca80ae9cc8791a72dab632090488093c5d0916f3`: [required CI](https://github.com/paperclipai/paperclip/actions/runs/42737676664) and [aggregate verify](https://github.com/paperclipai/paperclip/actions/runs/42737676664/job/129575813765) passed. Full rollup: 56 successes, four intentional skips and one advisory neutral result. Classic Snyk passed. [Greptile](https://github.com/paperclipai/paperclip/pull/20437#issuecomment-6756419093) and [Ponytail](https://github.com/paperclipai/paperclip/pull/20437#issuecomment-6756388693) both reviewed that exact head at 5/5. The one review finding is addressed and resolved; no open findings remain. Older partial CI and the initial 4/5 review remain nonfinal evidence. Hosted acceptance and measured wall-clock savings remain pending. ## Risks - Completed parent captures retain their scratch until the atomic publication completes. A 32-member cohort can retain up to four times the parent scratch of an eight-member cohort. Check actual parent capture, verification and restore sizes against current target filesystem capacity before selecting a larger cohort. Keep the existing resource guards. - Larger member arrays must not raise execution fanout. Every reader/reservation/final stage is bounded to eight and joins current peers before failure. Capture workers remain at most eight; child-verification workers remain two. - More than eight members require the exact same canonical physical root and complete source scope. A different root, new writer, changed source, recovery owner or journal still refuses through the existing owners. - Shared verification is limited to the held invocation fence. The separate fresh final pass remains required after all captures join. Different child sets and restart attempts do not share a local-byte proof. - The old saved membership and archive identities remain authoritative. Older tools fail closed on the larger bound. No schema, archive or provider format changes are included. ## Model Used OpenAI Codex, GPT-6 family, with repository tools and code execution. The runtime does not expose a more specific deployment identifier or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs: #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
1ca3e8ea2a |
fix(native): let a resume claimed after a restart write the run's existing log (#20427)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a native run is interrupted, Paperclip can persist a same-run resume: a later attempt continues the same run from its exact checkpoint > - If the server restarts before that resume is dispatched, the new server process claims it. The resume then reopens the run's existing log, and only the current controller may write it > - The resume claim took the lease owner but left the dead process's controller identity on the lease. The log writer check compares the boot id, so it refused, and the exact checkpoint resume failed as `run_log_writer_authority_lost` > - This pull request stamps the claiming process's identity on the resume claim, as restart recovery's own claim already does > - The benefit is that a resume persisted before a restart continues after it, instead of failing and needing a manual Retry ## Linked Issues or Issue Description Refs #19831. Storage chaos wave `w-20261110T003423-s9903` (local ST2, seed 9903), task t01 (codex C1, K1). The wave went 7/8; t01 was the one task that needed a Retry. - Run 279772b3 was SIGKILLed during its first turn. The restarted server resumed it, and the Codex snapshot failed ("cannot safely recover an ambiguous Codex turn start"). A same-run resume was persisted. - A second SIGKILL hit (00:35:37), and the next process dispatched the scheduled resume at 00:35:56. - The resume's run-log claim failed in `createRunLogWriterAuthority`: the coordinator lease still carried the killed process's `controllerBootId`. The exact checkpoint resume was refused (`run_log_writer_authority_lost`), and the run ended `adapter_failed`, needing a Retry. The wave had S3 latency injected, but the failure does not depend on it. The new test reproduces it on master with no storage faults. ## What Changed - `server/src/services/native-runtime/native-finalization-reconciler.ts`: `dispatchNativeSessionResumptions` writes the claiming process's controller identity (boot id, pid, host hash, process start) along with its lease owner and expiry. This matches the restart-recovery claim, so ownership proofs and the run-log writer check see the process that actually owns the resume. - `server/src/__tests__/native-session-resumption.test.ts`: a regression test for this case. ## Verification - `native-session-resumption.test.ts`: - A coordinator persisted for resume by a previous process (stale boot id), on a run with an existing object-store log, is dispatched. - The run-log writer check accepts the new claim, and the lease carries this process's identity. - The test fails on master with `run_log_writer_authority_lost`. - Regression: the native resumption, binding, recovery-deadline, restart-recovery, scheduled-resume, owner-release, runner-ownership and recovery suites pass locally. `tsc --noEmit` passes for `server`. ## Risks - The resume claim now also refreshes the controller pid, host hash and process start on the coordinator row. These describe the process that owns the lease, exactly as after a restart-recovery claim. A claim by the same process writes the same values as before. - No schema migration. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Nicky LeachandClaude Opus 5.5
|
232480d18f |
perf(runner): write preparation span rows to the run log in the background (#20426)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Before a native run starts its provider, the executor copies the preparation spans (queue, prepare phases, environment) into the run trace and the run log > - It records about 25 spans and waits for one run-log insert per span, one after another > - On a hosted test deployment, that step (`native.controller.span_persistence`, added in #20396) took a median of 0.28 s, and every task waited for it before the Runner started > - The run trace already has a way to write diagnostics without blocking the caller: `defer`, which `finish` drains before the root span closes > - This pull request writes those run-log rows through that path, in order, while the spans themselves are still recorded at once > - The benefit is about 0.25 s less before the provider starts, on every native run, cold or warm ## Linked Issues or Issue Description **What existing behavior does this improve?** The time between the end of heartbeat preparation and `native.session.execute` for native Runner tasks. **Subsystem affected** Server: native runtime tracing (`server/src/services/native-runtime/native-run-trace.ts`) and the native session executor. **Current behavior** `executeNativeSession` calls `trace.record` for each preparation span and awaits it. `trace.end` awaits `emitCompletedSpanEvent`, which inserts one `run.performance.span` row through `onEvent`. On a hosted test deployment, measured with the #20396 spans across 10 runs, `native.controller.span_persistence` took a median of 0.28 s: - 0.30 to 0.36 s for runs right after a computer wake - 0.17 to 0.25 s for runs on an awake computer It is the second largest step between preparation and the provider start (the median of that whole gap is 1.0 s). **Proposed behavior** The spans are still started and ended at once, with the same parents. The speed-budget bookkeeping still runs at once too. Only the run-log row is written later: one row at a time, in the order the spans ended, before the trace finishes. **Reason and benefit** The rows are diagnostics, and they already carry their own start time (`wallStartedAtMs`, `startOffsetMs`). Readers do not depend on the row order: the only reader in the repository, `summarizeStartupReuse`, looks for `runner.process.launch`, which this change does not defer. The provider can start without waiting for these inserts. **Breaking changes** None. ## What Changed - `native-run-trace.ts`: `end` and `record` take an optional `deferRunLog`. With it, `end` queues the run-log row on a per-trace chain registered with `defer`, so rows are written one at a time in order and `finish` waits for them. A row requested after the trace finished is dropped, as for other deferred diagnostics. - `native-session-executor.ts`: the preparation spans and the `environment.startup` scope use `deferRunLog: true`. - `doc/run-log-events.md`: note that native preparation span rows are written late, in order. - `native-run-trace.test.ts`: two tests. - Spans are recorded under their parent while every row write is blocked, and the rows are written in order before `task.provider_session.measured`. - A deferred row requested after `finish` is not written. ## Verification - `npx vitest run src/services/native-runtime/native-run-trace.test.ts src/services/native-runtime/native-run-trace-context.test.ts` (34 passed). Both new tests fail without the source change: the first times out waiting on the blocked write, and the second sees the row written. - `npx vitest run src/__tests__/heartbeat-preparation-gap-spans.test.ts src/services/native-runtime/native-session-executor.test.ts` (1321 passed) - `pnpm --filter @paperclipai/server typecheck` ## Risks - Low risk. The change is diagnostic only: OpenTelemetry spans and the speed-budget checks are unchanged. - The preparation rows can now land in the run log after the first Runner events. Readers already use the payload times, not the row order. - If the server process dies between the provider start and `finish`, the preparation rows of that run can be missing from the run log. The OpenTelemetry spans are not affected. - The deferred rows are written while the run continues, as budget-breach rows already are, so the sink sees concurrent writes from one run. This was already the case for deferred diagnostics. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, test runs, Honeycomb queries) and extended thinking. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
4608744977 |
fix(workspaces): preserve recovery for verified archived Git sources (#20436)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Durable workspace migration must retain saved Git files and pending native recovery. > - The archived-Git owner validates saved recipes, exact cleanup receipts, and surviving owned branches. > - Online recovery preservation currently accepts only the recipe variant. > - A valid cleanup receipt or owned branch therefore cannot migrate while an old native reference is still prepared. > - This pull request admits every verified archived-Git variant through the same recovery and finalization locks. > - Tasks stay done and keep their selected workspace, history, and pending provider obligations. ## Linked Issues or Issue Description Refs #20424, #20298, and #20418. The related cleanup work in #19812 does not cover this admission gap. **What happened?** Canonical inspection accepts the exact old commit of an archived Git workspace through a valid branch-deletion receipt or an owned surviving branch. Migration still reports `remote_copyback_unresolved` when a terminal native run retains a prepared reference. A read-only staging inventory reproduced this sole blocker for four such workspaces. Their original Git objects and canonical source proofs already exist. **Expected behavior** Preserve the verified Git baseline and every typed native/provider recovery obligation through the existing migration owner. Keep the task status, selection, history, archive state, and recovery records unchanged. Do not reopen the task or substitute the current default branch. **Steps to reproduce** 1. Save a clean archived Git workspace with an authentic cleanup receipt or a surviving owned branch, without a restore recipe. 2. Retain a terminal prepared native reference and its exact provider allocation and recovery action. 3. Remove the old task worktree. 4. Call `migrateRemoteWorkspace` online. **Paperclip version or commit** Reproduced on merged #20424, `915edcff49bb2cbacfbaee604b41e082e0397109`. **Deployment mode** Self-hosted server with PostgreSQL and configured object storage. ## What Changed - Change three conditions to accept a non-null canonical archived-Git source instead of requiring its recipe field. - Use the same typed recovery inspector, sorted native finalization locks, fresh reinspection, and source-task publication locks for all accepted variants. - Reuse the existing Git source validator, private scratch materializer, encrypted snapshot restore, journal, and recovery markers. No source-proof schema or provider protocol changes. - Extend the existing real fixture for cleanup receipts and retained branches. Verify exact old Git commits and unchanged task/recovery state. Refuse receipt revocation and branch movement during upload. ## Verification - Genuine RED on merged #20424: both new source-kind cases fail with `remote_workspace_migration_blocked:remote_copyback_unresolved,workspace_recovery_unresolved`. - GREEN: nine focused PostgreSQL/Git/encrypted-storage cases pass. They include both new variants, two actual upload-time source mutations, the existing recipe interrupted-journal resume, reference/provider mutations, and a competing native finalization owner. - Command: `pnpm exec vitest run server/src/services/remote-workspace-migration.test.ts -t 'no-recipe archived|unselected archived recipe|archived prepared source|competing native F owner'`. - All nine focused cases, server typecheck layers, module boundaries, and whitespace checks pass on exact published head `03065f97b31846a405a8911c30640ef62374e4a4`, after the clean rebase onto merged #20418. Required CI is now clean on this exact head:56 SUCCESS and4 expected SKIPPED checks, with no pending or failed checks. Fresh Greptile is5/5 with no actionable findings or unresolved threads. - Previous-release state: existing cleanup receipts, owned branches, prepared v3 references, and journal shapes carry forward unchanged. Saved snapshots and source identities retain their existing owners. Unrecognized or changed sources still refuse by the existing named errors. Migration does not declare remote files recovered or provider allocations stopped. - Hosted work in this PR was read-only source inspection and inventory. No staging import, source mutation, lifecycle reopen, or provider call was performed by this PR author. ## Risks The existing archived-Git validator now admits its other verified variants to recovery preservation. Incorrect source proof could publish incomplete authority. This PR keeps its exact commit/receipt/branch checks and full authenticated capture/readback. It also keeps all native ownership, reference, allocation, and publication checks. Missing or changed sources remain blocked, and pending remote data retains its original recovery obligation. ## Model Used OpenAI Codex, GPT-6, with reasoning, repository tools, and code execution. The runtime does not expose a deployment suffix or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Nicky LeachandClaude Opus 5.5
|
ff4ba7d319 |
perf(runner): run launch artifacts in place from a trusted read-only root (#20428)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Rust Runner (`paperclip-runnerd`) starts provider sidecars and proxies from qualified launch artifacts: a Node runtime and the scripts it runs > - Since #12699, Runner copies each artifact into a sealed memfd and checks its SHA-256 on every verification. This defends against a path swap or an in-place rewrite between the check and the exec > - On a hosted agent computer (4 vCPU), the copy and digest of the 127 MB Node takes about 0.26–0.30 s warm and about 1.2 s right after a wake. This is on the path from `session.open` to the sidecar spawn, and again on every warm `run.attach`. Each Runner also holds about 125 MiB of shmem for the copy > - On that computer, the artifacts live on the Runner ROM: a read-only erofs image that the deploy pins by digest. The threat that the memfd defends against does not exist there. On local hosts the artifacts are writable, so the memfd must stay > - This pull request adds an explicit, path-scoped, default-off `--trusted-launch-root <dir>`. Runner runs an artifact in place only when its open descriptor proves that it is on an immutable mount under that root. Everything else keeps the sealed, hashed snapshot > - The benefit is a shorter `session.open` and `run.attach` on agent computers, and about 125 MiB less shmem per Runner, with no change where the flag is absent ## Linked Issues or Issue Description **What existing behavior does this improve?** Launch-artifact verification in `paperclip-runnerd` (`qualified_launch.rs`). Its callers are the ACPX `verified_transport` (every `session_config`, including the park rebind) and the OpenCode and Cursor Cloud proxy launches. **Subsystem affected** Rust Runner (`packages/paperclip-runner/runner/crates/runner-core`). **Current behavior** Every verification opens the artifact, copies all bytes into a sealed memfd, and hashes them. Node is 127 MB. The cost is described in the Thinking Path. **Proposed behavior** When the host passes `--trusted-launch-root /opt/paperclip/runner` and the root passes its checks, Runner holds each admitted artifact open in place and executes it through `/proc/self/fd/N`. It does not copy or re-hash it. Without the flag, nothing changes. **Reason and benefit** See the Thinking Path. Refs #12699 (introduced the sealed memfd). Refs #20423 (overlap, see below). ## What Changed - `qualified_launch.rs`: new `TrustedLaunchRoot` and `set_trusted_launch_root` (process-global, set once at startup, like `--task-memory-cgroup`). - **Root checks at startup.** The path must be absolute, already canonical (no symlink or `..`), a directory, and on a read-only, exec-permitted erofs or squashfs mount. If a check fails, runnerd logs a warning and keeps today's behavior. - **Per-artifact checks, on the open descriptor.** After today's checks (regular file, not a symlink, not group- or world-writable, open-time identity), the descriptor must be a regular file on the root's device. Its kernel path (`readlink /proc/self/fd/N`) must lie under the root by path components, so `/opt/paperclip/runner2` does not match. Its mount must pass `fstatvfs` (`ST_RDONLY` set, `ST_NOEXEC` clear) and `fstatfs` (erofs or squashfs magic). If any check fails, Runner logs the reason and uses the sealed snapshot. - **Admitted artifacts** become `VerifiedProcessArtifact::in_place_on_trusted_root` (crate-private). The held file goes through the existing `inherited_artifact` path, so the exec is `/proc/self/fd/N` and `PAPERCLIP_VERIFIED_RUNTIME_EXECUTABLE` still names a descriptor that nested launches can reopen. - New `verify_assigned_executable_launch_artifact` never uses the root. `stdio_mcp.rs` now calls it, so host-assigned stdio MCP binaries always keep the sealed snapshot. - `paperclip-runnerd.rs`: parse `--trusted-launch-root <dir>`. Older runnerd builds ignore unknown flags, so a host can pass it before every Runner understands it. - The trusted path is Linux-only. On macOS and Windows, `TrustedLaunchRoot::open` always refuses. - `tests/runner-e2e/SECURITY.md`: documents how the Kimi and Pi provider-death fixtures behave under a trusted root. ### Decision: the trusted path drops the digest The trusted path does not hash the artifact, not even once per process: - The ROM image is pinned by digest at deploy time. - On the agent computer, the Launcher computes the expected `--acpx-sidecar-command-sha256` and `--acpx-sidecar-script-sha256` from the same ROM files immediately before it starts Runner. A re-hash of an immutable file can only repeat that result. - One hash per process would not help `session.open`, because each task starts a new Runner process. - The mount check is the guard: erofs and squashfs have no write path, so the bytes cannot change while the descriptor is open. ### Fixtures that assert the sealed memfd - Untrusted-mode assertions are unchanged. The fixtures do not pass the flag. - Kimi (`kimi-provider-fault.py` `checked_parent_descriptor`): with a trusted root, the parent descriptor is the ROM's Node file (one link). The existing linked-descriptor branch handles it: same inode as `/proc/<pid>/exe`, and the complete bytes hashed against `nodeSha256`. That branch never accepts the file as a memfd. The existing test `test_linked_descriptor_keeps_regular_single_link_checks_without_memfd_claim` covers it. - Pi provider-death admission binds `/proc/self/fd/3|7` to the native distribution snapshot's Node. The TypeScript ACPX closure copies that file whether or not a trusted root is set, so the flag does not affect it. ### Overlap with #20423 #20423 (draft) moves the same copy and hash to Runner boot and reuses the result. Where this flag is set (Cloud agent computers), there is nothing left to precompute, so #20423 is mostly redundant there. #20423 still helps hosts without a trusted root (local and sandbox Runners). This PR does not change #20423. ## Verification All runs are on Linux (aarch64, Rust 1.97.1, in a container): - `cargo fmt --manifest-path runner/Cargo.toml --all -- --check` - `cargo check --manifest-path runner/Cargo.toml --locked --workspace` - `cargo test -p paperclip-runner-core --lib -- qualified_launch trusted_root commonjs`: 12 passed. The new tests are: - `an_artifact_under_an_immutable_trusted_root_runs_in_place_without_a_digest`: trusted-root hit, with no memfd, the same inode as the source, and no re-hash. - `an_artifact_outside_the_trusted_root_keeps_the_sealed_snapshot`: a path outside the root is sealed, and a wrong digest still fails. - `a_sibling_with_the_root_as_a_name_prefix_is_outside_the_root`: `rom2/` does not match `rom`. - `a_root_that_is_not_read_only_falls_back_to_the_sealed_snapshot`: the real probe refuses a writable root at startup. A per-artifact mount failure falls back, and a wrong digest still fails. - `the_real_mount_probe_rejects_a_writable_filesystem` - `symlink_and_parent_escapes_from_the_root_fall_back`: a directory symlink out of the root, a `..` path out of the root, and a final-component symlink (still refused). - `a_root_must_be_absolute_canonical_and_a_directory` - `without_a_trusted_root_launch_artifacts_are_sealed_and_hashed`: flag absent, behavior unchanged. - `host_assigned_executables_ignore_the_trusted_root` - `trusted_root_artifacts_execute_the_held_file_and_reopen_it_for_nested_launches`: Runner execs from the held file, and a child re-execs `$PAPERCLIP_VERIFIED_RUNTIME_EXECUTABLE`. Both report the source file as `/proc/self/exe`. - `cargo test -p paperclip-runner-core --test stdio_mcp --test process_supervisor`: 4 and 8 passed. - Ad hoc, not committed: a privileged container with a real loop-mounted squashfs. The real `fstatvfs`/`fstatfs` probe admitted a `ro` mount and refused a `ro,noexec` mount. The unit tests replace only that mount probe, because CI cannot mount filesystems. - Not verified: a run on an agent computer with the real erofs ROM, and the latency and memory savings there. The Cloud Launcher change that passes the flag is a separate follow-up. ## Risks - **This change needs review from a security owner before merge.** It removes the sealed copy and the per-launch digest for admitted artifacts. That relies on the host's assertion that the root is a digest-pinned immutable image, plus the descriptor checks above. - **Misconfiguration fails closed.** A wrong, writable, `noexec`, non-canonical or missing root only loses the speedup, and runnerd logs a warning. There is no global "trust everything" switch, and Runner never selects a root from the filesystem it finds. - **Residual trust.** On an admitted file, Runner no longer detects a ROM image that differs from what the launch profile expected. On the agent computer, the Launcher derives the expected digests from the same ROM. Another host that passes the flag must pin its image the same way. An attacker with root in the guest (who could write the backing block device or remount) is out of scope; such an attacker can already control runnerd. - **Allowlist.** Only erofs and squashfs are accepted, because they cannot be written through any mount. A read-only bind mount of a writable filesystem is refused. A ROM that changes to another filesystem type will silently fall back to the sealed path (slower, still safe). - No migration. Behavior is unchanged when the flag is absent. Older runnerd builds ignore the flag. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, container builds). Extended reasoning was on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Devin FoleyandClaude Opus 5.5
|
5570debc5b |
fix(ui): stop reporting cancelled queries to Sentry (#20430)
React Query's CancelledError keeps the default Error name, so the name-based abort check never matched it. A query cancelled with revert: false (for example by the persisted cache when an issue read is denied or the account changes) reached the query cache onError as an "unknown" error and was sent to Sentry. Recognize it by class. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
18697b8d21 |
perf(workspaces): share cohort child-source verification (#20418)
## Thinking Path > - Paperclip must preserve agent workspace bytes during online migration. > - A cohort can contain several aliases of the same physical source. > - The importer verifies imported children before capture and again before publication. > - The immutable-object cache avoids repeated downloads but still recaptures local child bytes for every alias. > - This pull request shares the initial proof within the held cohort fence and checks fresh bytes once more after every capture joins. > - The benefit is less repeated local archive work while child mutations still block the whole cohort. > - Required browser CI twice reproduced an independent stale composer-mode bug. A focused regression proves that refreshed persisted task mode was ignored after mount. The state repair follows task data when there is no local override. Explicit local mode choices retain their own generation so a completed send cannot clear the next draft’s choice. ## Linked Issues or Issue Description Refs: #20311, #20387, #20401, #20404. **Current behavior** Two aliases recapture the same imported child's local bytes four times, even with one immutable object read. A larger alias cohort repeats this multi-GB work for each member. **Proposed behavior** Share the successful initial verification for the same physical root, Git roots and exact recorded child set or discovery mode. After all captures join, run a separate fresh final verification for each root and recorded child set. Keep every member's database and publication checks. **Reason and benefit** The local PostgreSQL fixture proves that two aliases now perform two child recaptures instead of four. Hosted timing improvement has not been measured. ## What Changed - Add invocation-local initial and final proof maps to the existing cohort child verifier. - Retain the existing canonical source-owner check before and after shared results. - Defer cohort final nested checks until all captures finish, and join every final reader before publication or fence release. - Keep singleton verification, saved journal schemas, snapshot IDs, archive bytes and per-member publication checks unchanged. - Extend the existing real PostgreSQL alias fixture with a child mutation during the last parent upload. - Repair the required CI planning-mode race with a nullable local override: absent override follows task data; a user selection belongs to the draft. A successful send or confirmed resend/receipt clears it only if no newer choice exists. Failed saves preserve the mode and restored message. Queued-edit backup retains whether the mode was explicit. Leave the browser assertion and timeouts unchanged. ## Verification - Canonical Node24 worktree preflight and plugin SDK prerequisites passed. Server TypeScript passed. - Genuine unchanged-source PostgreSQL RED: two aliases performed four local child recaptures instead of two (one failed, 22 intentionally not selected). GREEN: five source-sharing, last-upload child mutation, capture/cancellation join and actual outer-commit rollback controls passed (18 intentionally not selected; 66.53s). The actual fence-loss control also passed (one passed, 22 intentionally not selected; 38.13s). Both wrappers completed cleanup. - Migration source and tree bytes remain identical to locally qualified `466837cfe997924c3794fd5d0e96134cab1cd04c`; later commits only repair the composer and its tests. - Required planning browser CI originally failed, then failed its one unchanged-head retry, because the planning chip remained absent after opening-question dismissal. Those failures remain recorded ([original](https://github.com/paperclipai/paperclip/actions/runs/42723622165/job/129523785985), [retry](https://github.com/paperclipai/paperclip/actions/runs/42723622165/job/129534415590)). The final-head identical case passes in 30.5s, with all 34 shard cases passing ([final CI](https://github.com/paperclipai/paperclip/actions/runs/42731309796/job/129550549775)). Assertions and timeouts were not relaxed. - Composer regressions reproduced stale loaded mode, optimistic-save rollback, overwritten next-draft choices, and stale mode after confirmed resend/receipt. All 145 composer tests now pass (2.88s). Canonical UI TypeScript and token gates pass. The original planning browser case also passed locally (15.5s; 26.9s including setup). - The first final-head serialized-suite attempt was automatically cancelled at its unchanged 20-minute job limit. Grouped Turbo output did not expose an individual failing assertion; cause remains unknown ([retained attempt](https://github.com/paperclipai/paperclip/actions/runs/42731309796/job/129550550373)). One same-head diagnostic retry was a cache miss and executed all 419 heartbeat recovery tests successfully in 376.94s on the identical CI checkout ([retry](https://github.com/paperclipai/paperclip/actions/runs/42731309796/job/129561196792)). It is not classified as infrastructure or a known flake. - Final exact head `ba1f85f18085dd0e84c594ed0cb68d5bf817c93c`: all 61 checks terminal (56 success, four skipped, one neutral); required CI is green ([workflow](https://github.com/paperclipai/paperclip/actions/runs/42731309796)). Fresh Greptile is 5/5 with every review thread resolved ([review](https://github.com/paperclipai/paperclip/pull/20418#issuecomment-6754705323)). Focused commands: ```sh node --import ./cli/node_modules/tsx/dist/loader.mjs packages/db/src/test-postgres-command.ts pnpm exec vitest run server/src/services/remote-workspace-cohort-migration.test.ts -t "shares the merged child verifier|joins active capture peers|keeps every authority and activity unpublished" pnpm exec vitest run ui/src/components/task-chat/TaskChatComposer.test.tsx pnpm --filter @paperclipai/server exec tsc --noEmit pnpm --filter @paperclipai/ui typecheck pnpm check:token-gates ``` ## Risks - Reuse is limited to one invocation under the same complete source and finalizer fences. Different physical roots, Git roots, recorded child sets and discovery mode have different keys. No local-byte proof is stored durably or reused after restart. - Initial and final maps are separate. A child change during any parent upload must fail the fresh final verification before the atomic publication transaction starts. - Final readers all finish before an error can release the source fence. Per-member database child rows, source fingerprints, journals, keys and publication locks retain their existing checks. - This reduces repeated verification for aliases. It does not change capture concurrency, upload policy or immutable checkpoint validation. Hosted speedup remains unverified. - The CI repair must distinguish task data from explicit draft intent. Only a successful send with the same choice generation clears its override; a later draft selection and failed-save rollback retain it. Regression tests cover these transitions. It adds no styling, request, or persistence changes. ## Model Used OpenAI GPT-6 (Codex), with tool use and code execution. The runtime does not expose a more specific model identifier or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs: #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
915edcff49 |
fix(workspaces): preserve archived native recovery during migration (#20424)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Durable workspace migration must retain saved files and pending recovery work. > - An archived Git workspace can have an authenticated restore recipe and a terminal native run with pending remote files. > - Its task can select a different workspace. Reopening the archived workspace would change that task. > - The current migration owner refuses this source because it has no live local directory. > - This pull request uses the existing online recovery preservation owner for this exact archived recipe case. > - Migration retains the old task, provider allocation, recovery action, and native sync reference. ## Linked Issues or Issue Description Refs #20298, #20394, and #20418. Related cleanup work in #19812 does not cover this migration case. **What happened?** A historical archived Git workspace with a saved restore recipe and a terminal prepared native sync reference cannot migrate online. Admission reports `remote_copyback_unresolved` and `workspace_recovery_unresolved`. The old task can already select another workspace, so reopening it is not a safe migration path. **Expected behavior** Authenticate the archived files into private scratch. Preserve all typed native and provider recovery obligations in the existing migration journal and markers. Keep the original task selection, history, archive state, and restore flags unchanged. **Steps to reproduce** 1. Archive a Git worktree with staged, unstaged, and untracked files in a saved recipe. 2. Retain a terminal failed native run, its prepared sync reference, and its matching remote allocation and recovery action. 3. Remove the old local worktree and select a different workspace on the task. 4. Call `migrateRemoteWorkspace` without `writersStopped`. **Paperclip version or commit** Reproduced on `b2a7d322ed0bafda04a0b78ce14e4f25c8d5d170`. **Deployment mode** Self-hosted server with PostgreSQL and configured object storage. ## What Changed - Permit the existing online recovery owner to inspect recipe-backed archived Git sources. - Acquire the existing native finalization locks for those sources. Reinspect their exact proofs after lock acquisition and before publication. - Lock related source-task ancestry before publication. - Reuse the existing private Git restore, authenticated archive, journal, marker, and recovery-source contracts. Add no schema or provider calls. - Leave active sources and archived sources without a valid recipe on their existing refusal paths. - Include the shared UI correction from #20418. A nullable draft override follows fresh task data when the user has no local choice. Explicit choices increment a generation. Normal and recovered successful sends clear only their own choice; failed sends and newer choices remain intact. ## Verification - Unchanged master RED: the new historical-source fixture fails with `remote_workspace_migration_blocked:remote_copyback_unresolved,workspace_recovery_unresolved`. - Five focused real PostgreSQL, filesystem, Git, and encrypted-storage tests pass on Node 24.22.2. The success case loses an upload reply, resumes the same operation and snapshot, restores the staged/unstaged/untracked bytes, and verifies unchanged task history, lease, recovery action, finalizer, and recipe flags. - Recipe, prepared reference, and provider allocation changes during the actual upload each refuse publication. A competing native finalization owner refuses before capture. - Migration command: `pnpm exec vitest run server/src/services/remote-workspace-migration.test.ts -t 'unselected archived recipe|archived prepared source|competing native F owner'`. - All server typecheck layers, module boundary checks, and whitespace checks pass for the migration revision. The migration source bytes remain unchanged through the shared UI follow-ups. - Initial CI run 42727051535 exposed an existing planning-mode browser failure: shard 3/8 passed 33 cases but the refreshed task mode did not appear. The same issue occurred twice on #20418. A real local regression proved that initial cached data could leave the composer in its old mode. This path does not invoke migration. - Review then exposed genuine defects in the initial shared UI repair: an optimistic save rollback could reset a failed draft, and an earlier send could replace a newer draft choice. Recovered delivery also left a consumed choice pinned. All three have real RED→GREEN controls. The final shared fix uses an explicit nullable override, a choice generation, and one attempt-bound settlement owner for normal and recovered success, with no value-equality inference or durable payload changes. - Exact shared UI source now passes all 145 composer tests (2.88s), the original desktop/mobile planning browser case (15.5s; assertions and timeouts unchanged), token gates, and the canonical UI typecheck. The test harness uses the full persisted `IssueWorkMode` type, including `skill_test`, to fix the observed CI typing failure. - These UI results are reused from the identical shared source in #20418 through `ba1f85f18085dd0e84c594ed0cb68d5bf817c93c`. The unchanged planning browser pass is on the explicit-state model before the later settlement-helper extraction. The final receipt/resend controls, full composer suite, typecheck, and token gates include that extraction. They do not claim a local browser run on this entire combined branch. Fresh required CI and review qualify this PR head. - Previous-release state: existing recipe, v3 prepared reference, and journal shapes carry forward unchanged. A saved partial operation resumes with the same IDs. Migration does not certify remote file recovery or provider termination. Live provider recovery and hosted migration remain separate acceptance steps. ## Risks The change admits one more source class into an existing preservation path. An incorrect proof could publish incomplete file authority. The canonical recipe validation, full authenticated restore, native finalization locks, source reinspection, and publication checks remain mandatory. Pending remote data stays pending and retains its original allocation and reference. The shared UI repair updates only an unchanged draft mode; its regression checks preserve a local user choice. ## Model Used OpenAI Codex, GPT-6, with reasoning, repository tools, and code execution. The runtime does not expose a deployment suffix or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
3040448da5 |
fix(workspaces): refresh saved cohort source proofs on resume (#20429)
## Thinking Path > - Paperclip preserves agent work when a control-plane server is replaced. > - Online workspace imports save exact cohort, operation and snapshot identities for restart. > - A cohort proof also includes the complete current companion source scope. > - An independent companion import removes that companion from the local scope. > - The saved cohort then fails resume even when every saved member is unchanged. > - This pull request rechecks the current scope under fresh locks and refreshes its proof for all members in one transaction. > - The benefit is restartable online migration while independent work progresses. ## Linked Issues or Issue Description Refs: #19873, #20394, #20418. Related PR search found the original cohort owner, recovery-proof refresh and separate child-verification reuse. This fix addresses cohort scope freshness. **What happened?** An interrupted cohort rejects resume with `workspace_cohort_membership_changed` after a nonmember companion is migrated. The saved source proof includes companion workspace and task IDs. The exact saved members and their fingerprints can remain unchanged. **Expected behavior** The same saved cohort must resume after fresh admission. It must keep all member, operation and snapshot IDs. Changed members, active owners, changed journals and lost source locks must still refuse publication. **Steps to reproduce** 1. Start a two-member cohort and interrupt the second upload after the first archive is saved. 2. Import an archived nonmember companion through the existing canonical migration owner. 3. Resume the same two-member cohort. The original implementation refuses before it acquires the source fence. **Paperclip version or commit** Reproduced on master `3a5572c9e5`. **Deployment mode** Online administrative workspace migration with PostgreSQL and authenticated snapshot storage. ## What Changed - Admit a changed aggregate scope proof only when every saved member has a running journal and its current fingerprint matches both saved fingerprints. - Keep partial reservations on the original strict aggregate-proof path. Keep exact cohort and operation membership checks. - Run the existing source, lifecycle, recovery and finalizer admission before refresh. Lock member rows, inspect the current scope again, and compare every journal in one transaction. - Change only `cohortMigration.sourceProof`. Preserve all other metadata, identities, archive and recovery records. Abort the complete refresh on cancellation, a changed journal or a lost fence. - Extend the existing real PostgreSQL fixture with partial upload, canonical companion migration, refusal controls, transaction rollback, cold archive restore and replay. - Document the proof lifetime and resume boundary. ## Verification - Canonical Node24 worktree preflight passed. Plugin SDK build prerequisites passed. - Genuine RED on unchanged product: the real partial-upload and companion-import sequence fails with `workspace_cohort_membership_changed` (1 failed; 22 intentionally not selected; 10.44 seconds). - The identical positive passes after the fix (1 passed; 22 intentionally not selected; 9.87 seconds). - The complete focused cohort file passes: 23 tests in 33.33 seconds. Existing controls include changed member fingerprints, foreign and completed journals, active operations, peer cancellation, actual source-fence connection loss and outer-commit failure. - The new sequence also checks a held source lock, a held finalizer lock, cancellation before refresh, and an injected second-member update failure. The failed update rolls back every proof change. Both saved archives restore tracked and untracked work, and replay uses the same cohort. - After one final read-only preflight assertion, that complete new sequence passes again in 14.52 seconds with 22 existing cases intentionally not selected. Server TypeScript passes on the final files. Module boundaries and `git diff --check` pass. - Commands: `pnpm exec vitest run server/src/services/remote-workspace-cohort-migration.test.ts`; final case selection: `-t 'same uploaded cohort'`; `pnpm --filter @paperclipai/server exec tsc --noEmit`; `node scripts/check-module-boundaries.mjs`. - Initial local setup attempts remain nonpasses: sandbox PostgreSQL could not bind localhost, then the new archived companion fixture used a noncanonical repository folder. The corrected fixture uses the existing managed repository path. Neither setup failure is counted as the product RED. - Fresh [Greptile review](https://github.com/paperclipai/paperclip/pull/20429#issuecomment-6755862789) is 5/5 on exact head `6c640c4667b136e5416c1375f491729cf4bb9b49`, with zero inline threads. Required final-head CI is running. Hosted acceptance is not claimed by this PR. ## Risks - The old aggregate proof contains only a hash. Refresh therefore requires durable fingerprints for every saved member and repeats all current admission guards. A partially reserved cohort cannot use this path. - A stale proof must never hide a changed member or a new writer. Current source inspection, P/I/L/F locks, locked member rows, complete journal compare-and-swap and the existing capture and publication checks remain authoritative. - The all-member update must stay atomic. A database fault or cancellation rolls it back. The original cohort, operation and snapshot IDs remain fixed. - The change adds one bounded transaction only when a fully saved cohort's scope proof changes. It adds no recurring work, schema, provider, archive format, concurrency or deployment changes. ## Model Used OpenAI Codex, GPT-6 family, with repository tools and code execution. The runtime does not expose a more specific deployment identifier or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs: #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Nicky LeachandPaperclip
|
3a5572c9e5 |
Extend login PTY contracts for existing computers (#20410)
## Thinking Path
> - Paperclip manages agent execution across environments.
> - Environment providers own execution resources.
> - Codex and Claude already share the login PTY transport.
> - A login on an existing computer needs its own allocation and
deadline.
> - Input retries must retain their identity.
> - Codex also needs a fixed credential-file read; Claude already
returns its token through PTY output.
> - This change extends the shared contract and keeps both existing
provider parsers.
## Linked Issues or Issue Description
**Problem**
The existing login contract assumes a temporary sandbox. It cannot
describe a login allocation on an existing computer or export its fixed
credential result without generic execution.
**Proposed solution**
Add shared allocation purpose, command identity, PTY deadline, and input
identity fields for both Codex and Claude. Add one optional
credential-result hook for Codex's auth.json file.
**Alternatives considered**
A separate Claude login implementation would duplicate the current
setup-token parser and owner-bound credential writer. Claude continues
to use that existing path. The new artifact hook is Codex-specific
because Claude's setup-token result comes through the existing shared
PTY output stream.
## What Changed
- Export the login contract types from the main SDK entry point.
- Forward login purpose and command through plugin allocation and retain
them in lease metadata.
- Forward the PTY deadline and give each input write a stable identity.
- End a login route when the worker cannot confirm input delivery.
- Preserve the opening company scope for input, stop, and close
callbacks.
- Advertise and dispatch the optional fixed Codex credential-result
hook.
The allocation, deadline, input, output, and cleanup contracts serve
both providers. This PR does not enable either provider on an existing
computer. The dependent consumer PR connects admission, services,
credential completion, and UI for both providers.
## Verification
- Close-ordering regression: all 91 host worker-manager tests passed,
including concurrent close callers waiting behind a held plugin
acknowledgement after input failure. Server TypeScript check passed. All
current CI gates, ordinary review, and the selected follow-up review
pass on
|
||
|
DottaandClaude Opus 5.5
|
347d85fef3 |
perf(chat): index GitHub review lookups and share review-wait reads (#20400)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - GitHub chat endpoints let agents review pull requests. The server reads review state when it lists issues and when it updates GitHub checks. > - On a hosted instance, two GitHub review queries used the most total database time. One query ran about 660,000 times at about 33 ms. The other query ran about 10,000 times at about 1 s. > - The first query finds a source comment's chat link. No index covers `chat_message_links.comment_id`, so each call reads the full table. Issue lists also read the same review waits two times per request. > - The second query joins deliveries, links and runs on a JSON key. The planner walks the agent's run history in creation order and reads each run's context. > - This pull request adds four concurrent indexes, splits the run lookup into two small reads, and shares review-wait reads in one issue-list request. > - The benefit is less background database load, so request reads wait less. The results stay the same. ## Linked Issues or Issue Description **What existing behavior does this improve?** GitHub review state reads. These are the inherited review-wait lookup by source comment, and the review-check lookup of the run that a delivery's comment woke. **Subsystem affected** Server chat channels (GitHub review waits and checks), issue list attention maps, and the database schema. **Current behavior** - `readInheritedGitHubReviewWait` filters `chat_message_links` by `company_id`, `comment_id` and `direction`. No index has `comment_id`, so PostgreSQL reads the full table on each call. The run-origin check does the same lookup first. - An issue list calls the external review-wait read from the review attention map and again from the blocked-inbox attention map. The two maps read the same in-review issues. - `githubReviewCheckState` joins deliveries, links and runs with `context_snapshot->>'wakeCommentId' = comment_id::text`, sorts by `created_at desc` and takes one row. The planner reads the agent's runs in creation order and reads each run context. If no run matches yet, it reads all of them. The delivery filter on `normalized_event` and the link join on `delivery_id` also have no index. **Proposed behavior** The same rows come back from small index reads. Each issue's review wait is read one time per issue-list request. **Reason and benefit** These two statement families used about 33,000 s of database time in one cumulative statistics window. The database had about 50 concurrent active queries, and request reads waited behind this background work. **Breaking changes** None. The migration adds indexes only. ## What Changed - Migration `0456_chat_review_lookup_indexes` builds four indexes with `CREATE INDEX CONCURRENTLY IF NOT EXISTS`. It runs through the existing top-level statement path, which repairs an invalid index on retry: - `chat_message_links_company_comment_idx` on `(company_id, comment_id)` where `comment_id is not null`. - `chat_message_links_delivery_idx` on `(delivery_id)` where `delivery_id is not null`. - `chat_deliveries_github_automatic_delivery_idx` on `(endpoint_id, normalized_event->'githubAutomatic'->'context'->>'deliveryId')` where that value is not null. - `heartbeat_runs_company_wake_comment_idx` on `(company_id, context_snapshot->>'wakeCommentId')`. This index is not partial. The planner does not use expression statistics from a partial index. Without these statistics, it estimates many matches and keeps the creation-order walk. - `githubReviewCheckState` reads the delivery's comment IDs first. If there is a comment, it reads the newest same-agent run with one of these `wakeCommentId` values. If there is no comment, it skips the run read. The comparison stays text equality on `comment_id::text`. - `createExternalReviewWaitReader` shares one read per issue between the review and blocked-inbox attention maps of one issue-list request. The reader exists only for that request. - `doc/DATABASE.md` describes the indexes and the new read order. - `scripts/benchmark-review-lookup-indexes.ts` compares the pre-fix and new code with and without the indexes. `scripts/benchmark-review-lookup-index-migration.ts` times the migration on a large synthetic database. ## Verification Local SQL measurements on a synthetic fixture. They are not production timings. Environment: embedded PostgreSQL 18.1, baseline revision `02bfe71a6b`. Fixture: 50,000 links, 5,000 deliveries with about 2 KiB events, and 4,000 runs with about 12 KiB incompressible contexts. The wake run is in the middle of the run history. Each value is the median server execution time from `EXPLAIN ANALYZE` (2 warmups, 9 samples). The order was: no indexes, indexes, then no indexes again. | Caller and statement | Before (no indexes, two passes) | After | Shared buffers before → after | | --- | --- | --- | --- | | Inherited review wait, link by source comment (same SQL) | 2.72 / 2.36 ms | 0.076 ms | 929 → 12 | | Review check, wake run found | 31.9 / 21.9 ms | 0.29 ms (two statements) | 25,776 → 62 | | Review check, no wake run yet | 71.0 / 47.5 ms | 0.06 ms (two statements) | 34,861 → 12 | - Indexes alone do not fix the review check. The old SQL with the new indexes took 8.6 ms (run found) and 29.7 ms (no run), because it still walked the run history. The new SQL without the new indexes was neutral (23.5 and 35.1 ms). - Service wall times on this shared machine were noisy, so the table reports server execution time. - Correctness: the benchmark fails unless the inherited statement returns the same positional rows in each index state. The check service result and the final wake-run rows of the new code must also match the pre-fix module for both cases. - Migration timing, normal migration runner, synthetic data: 200,000 runs (1.8 GB with 8 KiB incompressible contexts), 500,000 links and 100,000 deliveries. First run 4.2 s; rerun 0.38 s. Separate builds: links by comment 1.5 s, run wake comment 1.1 s, deliveries 0.7 s, links by delivery 0.3 s. All indexes were valid. The run index was 8.3 MB. - New tests: - `server/src/__tests__/github-review-check-state.test.ts` checks that the newest same-agent run is selected. It also checks that other agents, other companies, uppercase IDs, non-string values, list-only `wakeCommentIds`, comment-less links and unknown deliveries do not match. It checks that the generated statements can use the new indexes. - `server/src/services/native-runtime/external-review-wait.test.ts` checks that the reader reads each issue one time, keeps concurrent callers correct, and rejects all waiting callers when a read fails. - Red/green: with the pre-fix `review-check-state.ts`, the behavior test passes and the index test fails. With this change, both pass. - Focused runs pass: `issue-blocker-attention`, `heartbeat-issue-liveness-escalation`, review-wait, review-check-state, external-review-wait, review-repair (106 tests), and `external-chat-wait.integration` (285 tests). `pnpm --filter @paperclipai/server typecheck` and `pnpm --filter @paperclipai/db check:migrations` pass. Commands: ```sh pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-review-lookup-indexes.ts --baseline-ref=02bfe71a6b pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-review-lookup-index-migration.ts ``` ## Risks - The migration builds four indexes concurrently. It does not block writes. On a large database, the run index reads every run context one time. The local run took 1.1 s for 1.8 GB from a warm cache. Cold storage on smaller compute can be slower. - The run index adds one entry for each run insert and for each run update that is not a heap-only update. Many runs have no `wakeCommentId`, so these entries hold a null key. - The review check now sends two statements instead of one when a delivery has a comment. Each statement reads only a few index entries. - When two matching runs have the same `created_at`, either can be selected. The previous query had the same tie behavior. - This change does not change the number of in-review issues that one issue list reads. The blocked-inbox map still reads all in-review issues of the company. A smaller scope needs a separate change to the liveness classifier. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) by Anthropic, in Claude Code with tool use (shell, file edits, local PostgreSQL benchmarks). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>canary/v2026.1110.0-canary.0 |
||
|
DottaandClaude Opus 5.5
|
326df52673 |
perf(api): rank Mine inbox candidates from the participation set (#20413)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - The Mine inbox reads `GET /api/companies/:companyId/inbox/mine`. Its
main query is `issueService.list` with "touched by me", "not archived by
me", six statuses including `done`, and `LIMIT 100` by canonical last
activity.
> - That query evaluates `touchedByUserCondition` (a five-way OR of
EXISTS) on every status-matching issue in the company. Because `done` is
included, the scan grows with company history, not with the user's work.
> - The unread badge already solved the same problem with a UNION of
indexed participation sources (#17973). The list path did not.
> - This pull request makes the Mine list rank page ids from that same
participation UNION, then load full rows (with the expensive select-list
subqueries) only for the 100 selected ids.
> - The benefit is that the main Mine query scales with the user's
participation instead of the company's issue count. Results stay
identical, and a new test proves it against the original scan.
## Linked Issues or Issue Description
**What existing behavior does this improve?**
The latency of the Mine inbox snapshot, `GET
/api/companies/:companyId/inbox/mine`.
**Subsystem affected**
Server: issue list service (`server/src/services/issues.ts`) and the
Mine route.
**Current behavior**
Measured on 2026-11-09: 280–570 ms server time on a dev instance and
1.9–3.2 s on staging. The list query scans every status-matching issue
(28k of 30k in the fixture below) to test participation, then sorts the
touched ones.
**Proposed behavior**
The list query reads only the user's participation set and computes the
row projection only for the page.
**Reason and benefit**
Inbox opens are frequent. The old cost grows with every completed task
in the company.
Refs #17973 (same UNION for the unread badge).
## What Changed
- Extracted the participation UNION from `unreadInboxIssueIds` into
`mineParticipationIssueIds()`. The unread badge uses it unchanged.
- New `IssueFilters.touchedCandidates: "participation"`. When it is set
with `touchedByUserId` (and there is no search and no
`oneChildPerParent`), `list`:
1. replaces `touchedByUserCondition` with `issues.id IN (<participation
UNION>)`. Each UNION member is one OR branch of the old condition, so
the set is the same;
2. runs a narrow `candidates` query (`SELECT id … WHERE <all other
filters> ORDER BY <same order> LIMIT/OFFSET`);
3. loads the full projection for only those ids (re-checking company,
visibility and the read condition), and restores the phase 1 order.
- Only the Mine route sets the flag. Every other `svc.list` caller keeps
the old SQL.
- A new Server-Timing phase, `inbox_issues_candidates`.
- `server/src/__tests__/inbox-mine-candidates.test.ts`: a
result-equivalence test.
- `scripts/benchmark-inbox-mine.ts`: a local benchmark that seeds a
large fixture, compares full payloads, and records `EXPLAIN (ANALYZE,
BUFFERS)` per query.
### Not changed, with evidence
- **Recomputed comment/activity maxima** (`lastActivityStatsForIssues`):
0.9 + 0.2 ms per call in the plans. Not worth a second code path.
- **Touched condition inside `userCommentStatsForIssues`**: 1.4–2.1 ms.
Not changed.
- **`liveDescendantCountMapForIssues`**: 12 ms at 30k issues and 40 ms
at 90k. It now costs about as much as the list query. It scans all live
runs and hashes all company issues. It is company-wide work, outside the
candidate selection, so I left it for a separate change.
## Verification
### Correctness
`inbox-mine-candidates.test.ts` runs the old scan and the new path on
the same data. It asserts equal ordered ids and equal full JSON
payloads. Fixture: 170+ touched issues (more than one page, with tied
timestamps) plus these edge cases:
- included: created by me, assigned to me only, user activity, my direct
comment, `done`, pending request addressed to me, archived then
re-activated by a newer outside comment, read state with a later agent
comment, a live child (live descendant count);
- excluded: archived with no newer activity, mention-only, read audit
only (`issue.read_marked`), uppercase entity id, channel-sourced
comment, agent-created with my user id, `cancelled`, hidden, harness,
plugin operation, private to another user (`issueReadSqlCondition`),
answered request, untouched issues, another company.
It also compares other shapes: no archive filter, `status=done` with the
full projection, `offset`/`limit`, ascending order, unrestricted read
condition, a second user, and a user with no participation. Mutation
check: removing the comment source from the UNION makes the test fail.
The benchmark also asserts identical full payloads for every sample of
both arms.
Focused suites (local): `inbox-mine-candidates`, `inbox-mine-routes`,
`authorization-privacy-query`, `issues-unread-inbox`,
`inbox-archive-routes`, `issues-assignee-unread`,
`issues-creation-visibility` — all pass.
### Performance (local SQL measurements)
Command: `pnpm --filter @paperclipai/server exec tsx
../scripts/benchmark-inbox-mine.ts --baseline-ref=18c3f4ef7b
--issues=<N> [--touch-density=<d>]`. Baseline =
|
||
|
DottaandClaude Opus 5.5
|
4a2eb920cf |
fix(storage): wait out a store outage on a run's critical path instead of failing it (#20299)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A cloud instance keeps run logs and evidence in object storage, so a run touches the store as soon as it starts > - In storage chaos wave `w-20261109T145628-s9902` (ST1: 75–176 s windows where every S3 call returns 503), only 1 of 8 tasks finished. A run starting or resuming during an outage failed within a second as `adapter_failed`, generic recovery burned its attempts seconds apart, the task was escalated to the board, and the user saw the raw S3 error > - This pull request treats a transient storage error as an infrastructure wait: the run is retried on a minute-scale schedule, rapid recovery is suppressed, the board shows it as working, and the run error says storage was unavailable > - The benefit is that a store outage of several minutes delays work instead of failing and stranding it ## Linked Issues or Issue Description Refs #19831. Storage chaos wave `w-20261109T145628-s9902` (master `610b54f551`, `--storage s3`, ST1). **Where the run fails.** At run start or resume, `run-output` `begin` opens the run log: `getRunLogStore().begin` → object run log store → object byte journal `claim` → S3 conditional PUT. The journal retries only 3 times within about 300 ms, so a 503 escapes and fails the run as `adapter_failed`. The async stack ends in the SDK middleware with no app frame. `log_store` stays null on the failed run, which pins the claim. **What followed.** - `failRun` released the issue into generic immediate recovery. Recovery retried 1–2 s apart inside the outage (t01: 14:58:33, :35, :59, 14:59:01), then stopped and opened a board-owned `stranded_assigned_issue` action. These are the open recovery actions on CHA-1, CHA-4, CHA-7 and CHA-8; a later Retry hit the outage again and escalated the same way. - The run error was the raw S3 text ("Injected storage outage"). **Slow store (ST2, `w-20261109T151946-s9903`).** With about 9.7 s of added latency per call, the same claim hit the journal's 15 s per-operation deadline (`AbortSignal.timeout(15_000)`). The run failed after about 36 s with `AbortError: Request aborted` and burned 3 runs the same way; 4 of 8 tasks needed a Retry. t02/t03 (Retry parked with `process_identity_missing`) are a separate native-recovery class and are handled in a follow-up. **Daytona storage chaos ST1 (`w-20261109T161202-s9912`, 1/8):** with S3 returning 503 for 72 to 154 s at a time: - Workspace restore failed at once (`remote_workspace_restore_failed: step=snapshot_download; storage=provider_error`), three times in 40 s, then `setup_failed`. It left an active recovery action behind. - Runs reattached after a restart died within 2 s on the run-log claim, then became reconciliation-required. - Setup failed on other storage reads. - 6 of 7 failures needed a UI Retry. So the rule here is one class rule: a transient store error on a run's critical path waits, it never fails the run. ## What Changed - New `server/src/storage/storage-unavailable.ts`: - `isStorageUnavailableError` reads the recorded storage failure diagnostics first, then the SDK error shape. Unavailable means 429/500/502/503/504, the S3 throttling and timeout names, network codes (`ECONNRESET`, `ECONNREFUSED`, `ETIMEDOUT`, `EPIPE`, `EAI_AGAIN`, `ENOTFOUND`), or an `AbortError` whose cause is a `TimeoutError` (a per-operation deadline). 403, 404 and 412 are not outages, and neither is an abort caused by the run's own Stop. - It defines the `storage_unavailable` code, the `storage_unavailable_wait` retry reason, the retry policy (60 s × 10 attempts, about 10 minutes) and the user-facing message. - `server/src/services/heartbeat/run-completion.ts`: - `failRun` classifies such a failure as `storage_unavailable` with the plain message. - Unless native recovery owns the run, it schedules the storage wait with `scheduleBoundedRetryForRun`, skips the interaction-continuation retry, and releases the issue with `suppressImmediateRecovery`. - `failRunSetup` uses the same policy for this code. - `server/src/services/heartbeat/retries.ts`: `storage_unavailable` joins the transient infrastructure codes. It gets its own retry budget and policy, like `runner_revision_mismatch`. - `server/src/services/task-interruption-presentation.ts`: `storage_unavailable_wait` is a resource wait, so the board shows the task as working with no Retry while it waits. - **One shared outage wait for the run's critical path** (`server/src/storage/storage-unavailable.ts`): - `retryThroughStorageOutage` retries a transient store failure with backoff (2 s doubling to 30 s) until that storage operation's outage has lasted 5 minutes, then gives the error back. The outermost wait of an operation records its first failure, and nested and parallel waits inside it share that start; no other request's success extends it. Shared probes and shared reads run under their own scope, so one run's Stop never cancels work another run waits on. A stop signal ends the wait, and non-outage errors are never retried. - Inside `waitOutStorageOutages`, every replayable S3 request waits instead of failing: get, head, delete, buffered and file puts, and multipart create, file parts and complete. A consumed upload stream is never resent, and requests outside the scope keep failing fast. - The scope covers run execution (`heartbeat.ts` `executeRun`: setup, workspace restore, run-log claim and the run itself), plus `loadWorkspaceSnapshotArchive` and `uploadWorkspaceSnapshot`, which copy-back and recovery workers use too. - The run-log claim retries as a whole, because the journal bounds each operation with a 15 s deadline. - Past the budget, the run takes the bounded `storage_unavailable` wait below. - The classifier follows the error's `cause` chain, so a restore failure wrapping an outage counts. `failRunSetup` checks storage before restore failures, so that case takes the storage wait instead of two quick retries and an escalation (the R6 leak). - `object-byte-journal.ts`: when its deadline expires, the journal rethrows the recorded storage error instead of a bare `TimeoutError`. ## Verification Red/green (each new test fails against master and passes with this PR): - `run-completion.test.ts` "waits a minute and suppresses rapid recovery when a run fails because storage is unavailable (storage chaos ST1)". An S3-shaped 503 records `storage_unavailable` with the plain message (no raw S3 text), schedules `STORAGE_UNAVAILABLE_RETRY`, releases with `suppressImmediateRecovery: true`, and skips the continuation retry. "keeps an ordinary adapter failure on the normal recovery path" guards the default. - `task-interruption-query.test.ts` "does not present an overdue storage_unavailable_wait wait as a failed attempt": working, `canRetry: false`. - `storage-unavailable.test.ts`: which errors count as an outage and which do not, including recorded diagnostics, a deadline-aborted call (ST2), and a Stop-aborted call that must not count. Static checks: - `tsc --noEmit` passes for `server`. Live verification: Phase 3 will rerun ST1 (seed 9902) with the MinIO fault proxy after merge. - `server/src/__tests__/storage-outage-wait.test.ts` runs against a local S3 that answers 503 for 3 s: - Inside the scope, a snapshot `getObject` and a run-log conditional PUT succeed once the store is back. - Outside it, a request fails fast. - A stop ends the wait. - Unit cases cover the budget, non-outage errors, and a `RemoteWorkspaceRestoreError` wrapping an outage. - `run-completion.test.ts`: a setup failure from a restore that wrapped an outage takes the storage wait (red before). - `object-byte-journal.test.ts`: a real `claim` whose head PUT hits its deadline is classified as a storage outage (red before). ## Risks - A storage outage now holds a task for up to about 10 minutes of automatic waits instead of escalating within seconds. A longer outage still ends in the existing board recovery. - A run whose failure is misclassified as storage-unavailable waits instead of using fast recovery. The classifier accepts only server-side or network failures from the storage SDK, never 4xx answers. - No schema change. - A run can now wait until a storage outage has lasted 5 minutes before it fails over to the bounded wait. It shows as working throughout, and Stop ends the wait. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
0116e45958 |
perf(db): index pending budget Stops instead of scanning all run results (#20391)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Budget enforcement stops agent runs when a budget hard limit is reached. The stop intent is saved in each run's `result_json` until run control delivers it. > - The budget worker, run control, and the idle-sleep check find these pending stops with a predicate on `result_json`. No index supports that predicate. > - Each check therefore reads every heartbeat run and its large result JSON from TOAST. On a hosted database, this statement family was among the most expensive: about 8,000 calls at about 1 s each, about 9,800 s in total. > - Pending stops are rare. The predicate is constant, so a partial index can contain only those rows. > - This pull request adds that partial index. The SQL that the server sends does not change. > - The benefit is that each check reads a few index entries instead of the full table. The database has more capacity for request work. ## Linked Issues or Issue Description **What existing behavior does this improve?** The check for committed budget Stops that are not delivered yet. The budget worker runs it each time it looks for work (`hasPendingBudgetEnforcement`), and again for each delivery pass (`listPendingBudgetCancellationScopes`). Run control and the idle-sleep check use the same predicate. **Subsystem affected** Budget enforcement in the server (`server/src/services/heartbeat/budget-cancellation.ts`, `server/src/services/budgets.ts`, `server/src/services/heartbeat/run-control.ts`, `server/src/services/idle-sleep-safety.ts`) and the `heartbeat_runs` schema in `packages/db`. **Current behavior** The query groups all rows of `heartbeat_runs` and reads `result_json->'startupCancellation'->'budgetEnforcement'` for each row. A limit of 1 does not help, because the group and sort occur before the limit. Cost grows with the run history, not with the number of pending stops. **Proposed behavior** A partial index, `heartbeat_runs_pending_budget_cancellation_idx`, contains only runs with a pending budget Stop. The planner uses it for all four callers. The results are the same. **Reason and benefit** Local SQL measurements show a 300x to 1,000x decrease in time and buffer reads (see Verification). **Breaking changes** None. ## What Changed - Added the partial index `heartbeat_runs_pending_budget_cancellation_idx` on `heartbeat_runs (company_id)`. Its predicate is identical to `pendingBudgetRunCancellationPredicate`. - Added migration `0455_pending_budget_cancellation_index.sql`. It uses `CREATE INDEX CONCURRENTLY IF NOT EXISTS`, and it is in the list of migrations that run outside a transaction, as `0436` is. - Moved the scoped run-control read and the idle-sleep check into exported builders in `budget-cancellation.ts`, next to the scope list query. The SQL does not change. The callers and the test now share the same statements. - Added a real PostgreSQL regression test. It checks that delivered, malformed, and missing intents are not pending. It also EXPLAINs the statements that the callers send (scope list with limit 1 and 100, scoped read, `EXISTS` check) and requires the new index in each plan. If a person changes a query shape, the server predicate, or the index predicate, the test fails. - Added `scripts/benchmark-budget-cancellation-scan.ts`. It captures the SQL that the real service sends, then compares the plans, timings, and complete rows with and without the index. - Updated `doc/DATABASE.md`. ## Verification Red/green regression test: - Before the migration, the new test failed: the plan was `Seq Scan` with the predicate as a filter. - After the migration: `npx vitest run src/services/heartbeat/budget-cancellation.test.ts` passes (17 tests). - Also passed: `src/__tests__/budgets-service.test.ts`, `src/__tests__/idle-sleep-diagnostics-postgres.test.ts`, `src/services/budget-resume-continuation.test.ts`. - `pnpm --filter @paperclipai/db check:migrations`, and typecheck of `@paperclipai/db` and `@paperclipai/server` pass. Performance and result equality (local SQL measurements only): ```sh pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-budget-cancellation-scan.ts --runs=20000 --payload-kib=16 [--reverse] ``` - Fixture: embedded PostgreSQL 18.1. 20,241 runs in 2 companies, with incompressible result JSON of about 16 KiB (table 379 MB). 40 pending stops in 6 scopes, 200 delivered or malformed decoys, and 1 cross-company intent. - The SQL is the same in both treatments. The script toggles only the new index. 2 warmups and 7 samples per query. Rows are compared outside the timed part. Ordered queries must match in order. The scoped run-control read has no `ORDER BY`, so it is compared as a multiset. | Query (caller) | Baseline median | With index median | Shared buffers | | --- | --- | --- | --- | | `listPendingBudgetCancellationScopes(db, 1)` (worker `hasPending`) | 688 ms | 2.1 ms | 243,500 → 1,158 | | `listPendingBudgetCancellationScopes(db)` (delivery) | 799 ms | 1.4 ms | 243,500 → 1,158 | | Scoped pending runs (run control) | 357 ms | 1.1 ms | 121,835 → 322 | | `EXISTS` pending (idle-sleep check) | 440 ms | 0.37 ms | 240,560 → 2 | - Reverse order (index first) gave the same result: 524 ms → 1.6 ms, 464 ms → 1.4 ms, 306 ms → 1.0 ms, 306 ms → 0.42 ms. - Plans change from `Seq Scan` (or a broad bitmap scan) to `Index Scan` or `Index Only Scan` on the new index. - All result rows are equal in every run. Migration duration: - 20,241 runs (379 MB): index build 2.0 s to 3.9 s. - 200,241 runs with 8 KiB result JSON (2.1 GB): index build 5.6 s. With this table size, the four queries change from 1,148 / 879 / 514 / 4,392 ms to 1.1 / 1.2 / 0.85 / 8.3 ms. - The build is concurrent, so it does not block run writes. ## Risks - Migration: the build reads every run's result JSON one time. It is concurrent and resumable (`IF NOT EXISTS`). The executor repairs an invalid concurrent index on retry. - Write cost: run writes evaluate the index predicate. The index contains very few rows, so the extra cost is small. - Predicate drift: if a later change edits only one of the two predicates, the planner cannot use the index. The new regression test detects this. - The hosted timings are not measured yet. The local numbers come from a synthetic fixture. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, local PostgreSQL). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
ab80fea62f |
perf(recovery): probe run bindings per comment in the lost-wake check (#20412)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The ownership reconciler finds user comments whose wake a restart lost, and it saves a wake for them (plan invariant 4) > - Its lost-wake check matched a task's runs with one OR across the legacy context ID and the native issue column > - With that OR, PostgreSQL can join runs only by company. It then tests each recent comment against every run of its company and reads each run's context > - On a hosted staging database, this statement used about 23 s per call on average (384 calls, about 8,900 s in total). One call ran for 35 s. Other requests waited behind it > - This pull request reads the check in three stages and probes the two run bindings separately with existing indexes > - The benefit is the same result with bounded, index-driven work for each recent comment ## Linked Issues or Issue Description **What happened?** The reconciler's user-comment lost-wake check was one of the most expensive statements on a hosted staging database. Query statistics showed 384 calls with a mean of about 23 s. One active call was seen at 35 s. The database compute was busy, and short authentication reads waited behind long statements. The statement matched runs with `(context_snapshot->>'issueId' = issue OR native_issue_id = issue)` in three subqueries. The OR gives PostgreSQL no join key except `company_id`. A local plan on synthetic data showed the cost: a hash join of 273 comments against all 38,002 runs, which took 1.0 s of a 1.3 s statement. The work grows with recent comments multiplied by the company's run count, and each test reads a run context. **Expected behavior** Each recent comment uses a small number of indexed probes. The check returns the same comments in the same order, with the same limit and grace logic. **Steps to reproduce** 1. Seed a company with a deep run history and a few hundred recent user comments. 2. Let some recent comments have no wake (for example, after a restart lost the wake of a steered comment). 3. Run the check. PostgreSQL can pick a hash join on `company_id` with the OR as a join filter. With nested loops off, the old statement takes seconds. The local benchmark below shows this. **Paperclip version or commit** Baseline: `02bfe71a6b`. **Deployment mode** Server with PostgreSQL. Local proof uses disposable PostgreSQL 18.1. Production and staging use hosted PostgreSQL. Related: the earlier change that split native and legacy bindings for other ownership reads (see the "Execution recovery query performance" section in `doc/DATABASE.md`). A search found no open pull request for this check. ## What Changed - `checkUserCommentsWithoutWake` reads in three materialized stages: 1. Recent user comments on assigned, non-conversation tasks that no wake request carries. This stage keeps the same issue, status and agent-closed rules. 2. Of those, comments with no live or later run and with at least one earlier run. 3. Of those, comments that no live run acknowledged in an item event (the steer check). Only this stage reads event payloads. - Each run lookup is two probes: one on the legacy context ID and one on the native issue column. Both use existing indexes. `exists (A or B)` becomes `exists (A) or exists (B)`, which is logically the same. - Each probe ends with `offset 0`. This keeps it a per-row index lookup. Without it, PostgreSQL can build a hashed subplan over all runs, or a hash join over all item events of the company. The repository already uses this fence in `execution-blocker.ts`. - The final `order by c.created_at limit N`, the projection, the grace window and the reported reason do not change. - Add `scripts/benchmark-comment-wake-check.ts`. It loads the check from an explicit baseline commit and from the working tree, and it compares the complete service result and the complete positional SQL rows before it records time. - Add a regression test for each run binding on its own: native only, legacy only, live run, later run, steered run, an acknowledgement before the comment, a log-only mention, a run that ended before the comment, and a run in another company. - Document the check and the benchmark command in `doc/DATABASE.md`. - No schema change, index, migration, or new startup work. ## Verification - `pnpm exec vitest run server/src/__tests__/ownership-reconciliation-report.test.ts --maxWorkers=1`: 34 passed. This includes the existing lost-wake, restart, steered and agent-closed tests and the new binding test. - `pnpm --filter @paperclipai/server typecheck`: passed. - `pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-comment-wake-check.ts --baseline-ref=02bfe71a6b --samples=7` Benchmark setup: PostgreSQL 18.1 (embedded, local), `shared_buffers` 128 MB, `work_mem` 4 MB. All data is synthetic, with md5-hex filler so contexts and payloads do not compress well. There are two companies per fixture. Two warmup rounds, then 7 timed samples for each arm, in alternating order. Each sample runs the real check function. Before any timing, each fixture asserts that the complete check result and every positional SQL row are equal for baseline and treatment, and that the fixture reports at least one owed comment. Every timed sample also asserts the same result. | Fixture | Runs / recent comments / owed | Baseline median | Treatment median | Speedup | Hash-join plan, baseline → treatment | | --- | --- | --- | --- | --- | --- | | small | 2,541 / 50 / 6 | 8.5 ms | 2.6 ms | 3.2x | 19 ms → 3 ms | | representative | 75,502 / 750 / 75 | 26.4 ms | 23.3 ms | 1.1x | 2,262 ms → 17 ms | | lost steer wakes | 38,002 / 750 / 75 | 454.7 ms | 66.9 ms | 6.8x | 2,717 ms → 68 ms | | deep history | 150,750 / 1,125 / 114 | 26.9 ms | 18.8 ms | 1.4x | 4,006 ms → 34 ms | | busy hour | 128,750 / 5,625 / 200 (limit) | 79.2 ms | 47.4 ms | 1.7x | 20,047 ms → 75 ms | | long steered runs (stress) | 19,100 / 600 / 60 | 626.4 ms | 457.4 ms | 1.4x | 5,871 ms → 502 ms | - The table above used baseline `02bfe71a6b`, the branch base at that time. After the rebase, a repeat run with baseline `8004ad069e` (the new merge base, with the helper guard fixed to run from the repository root) gave the same picture on a busier machine: small 15.4 → 9.1 ms, lost steer wakes 1,733 → 139 ms, busy hour 103 → 65 ms, all with equal results. - The "lost steer wakes" fixture shows the slow plan with default planner settings. PostgreSQL hashed all runs by company and tested 273 comments against 38,002 runs. - The last column runs each statement once with `enable_nestloop = off`. This makes the company-only hash join that the old statement allows visible, and it shows that the new statement has no such plan. The result is checked as equal in this mode too. Staging's 23 s mean matches this growth: recent comments multiplied by the company's run count. - Shared buffers fell in every fixture except "small" with default settings (for example, busy hour 176,416 → 89,803). - Neutral path: in the "long steered runs" stress fixture, every steered comment lost its wake. These comments correctly reach the steer check. That check must read the payload of each item event after the comment in the run that was live, so it is the remaining cost. In normal operation a comment sent to a live run has a deferred wake that carries it, so stage one removes it. ## Risks - Low risk. The predicate is the same conjunction of conditions. Only the evaluation order and the probe shape change, and the benchmark asserts complete-row equality on six fixtures and in both planner modes. - Rows with the same `created_at` have no defined order, as before. The benchmark uses distinct times so positional comparison is meaningful. - `offset 0` stops PostgreSQL from turning these probes into joins. If a task someday has a very large number of runs, each probe still uses the company and issue indexes. - Upgrade safety: this change reads saved run, wake and event state only. It changes no persisted shape, run identity, or saved session, so state saved by the previous release carries forward unchanged. A cutover test is not applicable. - Not verified on the hosted database. The local proof uses synthetic data, and I did not capture a hosted plan. Confirm with a fresh query statistics window after deployment. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell, file editing, local PostgreSQL) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
60a60a0cd1 |
fix(opencode): attribute unreadable tool-result images to the tool (#20399)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The Paperclip runner drives OpenCode agents. It sends their
OpenRouter and Paperclip-connection model calls through a local provider
proxy.
> - An agent often takes a screenshot of its own work and reads it to
verify the result.
> - OpenCode 1.18.34 cannot put images in OpenAI-compatible tool
results. It moves them into a synthetic user message ("Attached media
from tool result:"). For a text-only model, it replaces the image with
"ERROR: Cannot read image (this model does not support image input).
Inform the user."
> - The model reads that text as a message from the person. In a chat
eval, a DeepSeek worker redid its verification for about 5 minutes,
registered its deliverable twice, and ran out of the 15-minute attempt
clock.
> - This pull request moves that notice into the tool result that
produced the image, worded as a runner note, and removes the synthetic
user message.
> - The benefit is that a text-only agent sees a clear tool outcome and
continues its work. It does not mistake the notice for the person.
## Linked Issues or Issue Description
**What happened?**
An OpenCode agent on `openrouter/deepseek/deepseek-v4.1-flash-1017` read
a PNG screenshot that it made. The next model request contained a
user-role message with the text "Attached media from tool result: ERROR:
Cannot read image (this model does not support image input). Inform the
user." The model reasoned "The previous input from the user was ..." and
repeated finished work.
**Steps to reproduce**
1. Run OpenCode 1.18.34 with an `@ai-sdk/openai-compatible` provider and
a model that has no image input.
2. Have the model call the `read` tool on a PNG file.
3. Inspect the next chat-completions request. The last message is the
synthetic user message above.
**Expected behavior**
The model must get the notice as part of the tool result, not as a
message from the person.
**Environment**
Paperclip runner, OpenCode 1.18.34 (qualified pin), OpenRouter and
Paperclip-connection model routes.
## What Changed
- Add
`packages/paperclip-runner/src/drivers/opencode/tool-media-attribution.ts`.
It finds the OpenCode synthetic attachment message only when it directly
follows tool results. It moves each "Cannot read ..." or "empty or
corrupted" notice into the last tool result of that step as a runner
note. It keeps readable media in the synthetic message. It removes the
message when no media remains.
- The OpenCode provider proxy applies this rewrite before it forwards a
request. It re-serializes the body only when something changed.
- Messages written by a person are never changed. A person who attaches
an image to a text-only model still gets "Inform the user", which is
correct for that case.
- Add unit tests for the rewrite and an integration test through the
real provider proxy.
## Verification
- Red: the new proxy test failed before the change (`expected ... to
have a length of 4 but got 5`).
- Green: `npx vitest run
src/drivers/opencode/opencode-server-driver.test.ts
src/drivers/opencode/tool-media-attribution.test.ts` (82 tests pass).
`tsc --noEmit -p packages/paperclip-runner` is clean.
- Real binary: I ran the real OpenCode 1.18.34 binary with a mock
OpenAI-compatible server and had the model read a PNG. The captured
request has exactly the message shape that the test fixture uses.
- `pnpm verify` returned exit 2 (test files outside the resolved
graphs). Required CI supplies broad coverage.
- A paid live run of
`chat-features.runner-opencode.local.task-coordination.delegate-with-context`
will follow in a PR comment.
## Risks
- Low risk. The rewrite matches the exact OpenCode 1.18.34 strings. If a
future OpenCode version changes them, the rewrite does nothing and the
old behavior returns. It does not fail the request.
- The rewrite is deterministic, so prompt-cache prefixes stay stable
across requests.
- No persisted state changes. OpenCode history stays as it is. Sessions
saved before this change also get the rewrite when they resume.
- Providers that do not use the runner provider proxy are not covered by
this change.
## Model Used
- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use (shell, file edits, test execution).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandPaperclip
|
88186fe578 |
fix(workspaces): preserve unmaterialized repository intent during migration (#20416)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Workspace migration preserves saved files when a web server becomes replaceable. > - A repository request records intent before any checkout exists. > - Online migration currently requires that request to finish, even when preparation needs credentials and has saved no repository files. > - This pull request preserves that database intent while migrating the existing workspace files. > - Normal admission still owns repository acquisition and authorization after restore. ## Linked Issues or Issue Description **What happened?** Online workspace migration refuses a pending repository request, or a request marked `needs_repair` with `repository_preparation_failed`, even when its contained checkout path does not exist. This can force an unrelated agent turn and credential acquisition before existing files can be preserved. **Expected behavior** Preserve an absent, unpinned repository request unchanged. Migrate the existing files under the normal source locks. Prepare the repository only on a later authorized admission. **Steps to reproduce** 1. Create a task-scoped local workspace with saved files and an absent repository checkout. 2. Keep its repository request pending, or let canonical preparation fail before checkout publication. 3. Attempt online workspace migration. Before this fix, it refuses with `repository_preparation_unresolved`. **Paperclip version or commit** Reproduced against master `8004ad069e`. **Deployment mode** Self-hosted control plane with PostgreSQL and local legacy workspace files. The regression uses real local PostgreSQL and independent filesystem restore. Related migration work: #20371. No duplicate open unmaterialized-request fix was found in the GitHub search. ## What Changed - Accept unchanged pending requests and canonical preparation failures only when no pin or branch exists and the contained checkout path is absent. - Require a proven online source or the existing offline writer-stop contract. - Leave companion workspace requests under their existing strict blocker. Classify the current workspace request with its filesystem check. - Extend existing migration and cold-admission fixtures. Repository rows retain their catalog, requested ref, failure state, timestamps, and identity. ## Verification - RED: the real PostgreSQL/filesystem regression produced two `repository_preparation_unresolved` failures for online pending and failed requests. Offline pending passed. Duration: 24.95 seconds. - Local server TypeScript check passed with Node 24 and pnpm 9.15.4. - GREEN on final head `eeaea6134a56397ca8c12ed8f7009a19c41d79b1`: 18 focused cases passed across the existing migration and cold-admission suites in 77.48 seconds. This covers offline pending intent, online pending/failed intent, active preparation and partial checkout refusals, strict companion requests, publication races, real archive restore, and later normal repository preparation. - The first GREEN attempt reached successful cold preparation, then failed one inherited assertion that expected the old failure code after success. The assertion now expects the canonical owner to clear it. The migration row-equality checks had already passed; no production change was needed for this assertion failure. - Final-head [PR workflow 42723571639](https://github.com/paperclipai/paperclip/actions/runs/42723571639) completed SUCCESS. The [final `ci / verify` gate](https://github.com/paperclipai/paperclip/actions/runs/42723571639/job/129531238991) passed at 2026-11-10 00:08:04 UTC. The full rollup contains 55 successful checks and 4 intentional skips, with no pending or failed checks. Greptile reviewed exact head `eeaea6134a56397ca8c12ed8f7009a19c41d79b1` at 23:52:42 UTC with 5/5, a successful check, and no inline threads. The optional `pnpm verify --explain` selection was invalidated when the assertion commit landed during selection; no verification pass is claimed from it. Focused local tests and required CI supply the verification scope. - The first local test invocation could not resolve the plugin SDK before its local build. It ran no tests. The SDK was then built and the RED regression ran successfully as a failure proof. ## Risks - The failed-request exception is limited to `repository_preparation_failed`. Unknown failures remain blocked. - Active preparation, a pin, a branch, a partial checkout, unsafe paths, and unresolved companion requests still block migration. - Migration does not edit repository intent or grant repository/task access. A restored request can still fail normal preparation if its current authorization or source is unavailable. - Offline migration retains its existing writer-stop and maintenance checks. Online migration retains its source, task, lifecycle, and publication fences. - This patch changes internal migration admission. No schema, public API, user command, or documentation contract changes. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and tool use. The exact serving model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
b2a7d322ed |
test: preserve runner commit stall acknowledgement invariants (#20383)
Assert that stalled event commits do not advance acknowledgement or committed state. Clarify the limits of historical failure attribution. Preserve the production implementation already merged in #20385. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
50df9a19b9 |
perf(inbox): read only the execution workspaces Mine rows reference (#20388)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Mine inbox is the first page a board user opens to see the tasks that need them > - The page loaded the company's full execution-workspace summary list and kept task rows hidden until that list arrived > - On a company with thousands of workspaces, that read checks each workspace row against the privacy predicate. It was the slowest request on the page > - The inbox needs workspace names only for the workspaces its rows reference, and only when the view shows workspace data > - This pull request adds an `ids` filter to the summary read, makes the inbox request only referenced workspaces, and stops rows from waiting for workspace data when the view does not show it > - The benefit is that Mine rows appear when the task data arrives, and the workspace read becomes a small bounded query ## Linked Issues or Issue Description **What existing behavior does this improve?** The Mine inbox (`/:companyPrefix/issues?view=mine`) shows its task rows late on companies with many execution workspaces. **Subsystem affected** Board UI inbox and the execution-workspace summary API. **Current behavior** The page calls `GET /api/companies/:id/execution-workspaces?summary=true` with no filter. On a large company this returned 8,758 rows (2.5 MB) and took 12.3 s on the server. The reveal gate included this read, so rows appeared at 13.4 s although `inbox/mine` arrived at 1.6 s. **Proposed behavior** The inbox requests summaries only for the workspaces its rows reference. Rows appear when the Mine task data arrives, unless the view shows workspace data. **Reason and benefit** The workspace read becomes a small bounded query, and it leaves the critical path for the default Mine view. **Breaking changes** None. The `ids` parameter is optional and additive. ## What Changed - `packages/shared`: add `executionWorkspaceSummaryIdsQuerySchema` (comma-separated UUIDs, 1–100 distinct values) and `EXECUTION_WORKSPACE_SUMMARY_IDS_MAX`. - `server`: `GET /companies/:companyId/execution-workspaces` accepts `ids`. Bad input returns 422. The service adds `inArray(execution_workspaces.id, ids)`. The company scope and the actor read condition still apply, so privacy does not change. - `ui/src/api/execution-workspaces.ts`: `listSummaries` accepts `ids`. New `listSummariesByIds` reads ids in chunks of 100 and concatenates the results. - `ui/src/pages/Inbox.tsx`: the workspace query is keyed on the sorted distinct `executionWorkspaceId`s of the rows the inbox holds (Mine, touched, context and remote search issues). It uses `keepPreviousData` and runs only when isolated workspaces are on and at least one id exists. - Reveal gate: rows wait for the workspace read only when workspace data changes what the rows show. This is workspace grouping, a visible Workspace column, or an active search (workspace names are searchable). Otherwise rows reveal without it. - Tests: route ids pass-through and 422 cases (non-uuid, empty, separators only, more than 100); service test for the id filter with company scope and the read condition; UI api chunking test; Inbox tests for "rows reveal while the workspace read is pending when no workspace column or grouping is active", rows waiting when the Workspace column is visible, and "shows the resolved isolated workspace name" with the id-filtered call. Follow-ups (not changed here, they still read the unfiltered list): - `ui/src/pages/LegacyInbox.tsx:772` - `ui/src/components/IssuesList.tsx:985` - `ui/src/components/LegacyIssuesList.tsx:888` - `ui/src/components/NewIssueDialog.tsx:570` (filtered by project and reuse, but not by ids) ## Verification Focused checks on this revision: - `cd server && npx vitest run src/__tests__/execution-workspaces-routes.test.ts src/__tests__/execution-workspaces-service.test.ts -t "summar|ids"`: 14 passed (embedded Postgres ran the service test). - `cd ui && npx vitest run src/api/execution-workspaces.test.ts src/pages/Inbox.test.tsx`: all passed. - `pnpm --filter @paperclipai/server --filter @paperclipai/ui --filter @paperclipai/shared typecheck`: passed. - `pnpm check:token-gates`: clean. - `pnpm verify` selected 2,481 Vitest files (the shared validator change reaches most suites). I stopped it at a 25-minute limit on a heavily loaded shared host. Before the limit, three server tests outside this change failed: `agent-commentary-skills` (skill mount), `agent-config-read-access` (24.8 s timeout) and `agent-directory-working-copies`. Required CI supplies the full coverage. - Review fix `fc3f82dae0`: when workspace data is visible, the reveal gate also waits while `keepPreviousData` supplies the previous id set's summaries. A new test switches Mine → Recent to a new workspace and checks the Recent row stays hidden until its workspace loads. It fails without the change. - Follow-up commits `bf34f84823` and `8fc743b5ca` add a 100-row Mine test with a counted workspace id scan. It fails with the old `= []` defaults and passes with the stable `NO_ISSUES` default. Performance (local, synthetic fixture): - Instance: isolated worktree instance, `authenticated` mode, signed in as a non-admin company member so the real workspace privacy predicate runs. Production UI bundle (`PAPERCLIP_UI_DEV_MIDDLEWARE=false`). Headless Chromium 1440×900, a new browser context per run (cold HTTP cache and cold query cache). macOS, Apple Silicon, with other local sessions running. - Fixture: 8,750 execution workspaces (each with a source task and a privacy-source key), 38,750 tasks, 150 Mine tasks; the Mine snapshot returns 100 rows that reference 100 workspaces. - Finish event: the first `[data-slot="task-row"]` is in the DOM. | Measurement | Before (`02bfe71a6b`) | After | | --- | --- | --- | | Cold time to rows, median of 7 | 1,209 ms (1,046–1,312) | 702 ms (567–1,209) | | Workspace summary read, server wait (time to first byte), median of 7 | 496 ms unfiltered, 2.0 MB | 18 ms with 100 ids, 22 KB | The difference of medians is 507 ms. This is close to the unfiltered read's wait on this fixture (about 550–590 ms in the browser runs). The runs were sequential batches, not alternating pairs. Limits: this fixture does not reproduce the 12.3 s server time seen on the development instance. Here the unfiltered read cost 0.5–1.7 s and grew with the company's task count. The browser gain on the original instance should be close to that instance's unfiltered read time, but I did not measure it there. ## Risks - Low risk. The `ids` filter only narrows the existing query. It does not change the read condition or the company scope. - A row whose workspace is not readable by the actor shows no workspace name, as before. - When the Workspace column or grouping is visible, rows still wait for the workspace read. That read is now small. - When the id set changes, the query refetches all chunks. Before a view first reveals, rows wait for the new summaries if workspace data is visible. After that, the previous names stay visible while a refresh loads. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edit, headless browser). Extended reasoning on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
1c868b86be |
fix(server): tell agents who owns a plugin-managed routine and where it is changed (#20409)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A plugin can create and own routines ("plugin-managed routines").
The routine row shows the plugin in `managedByPlugin`.
> - An agent reads routines through `GET` and `PATCH
/api/routines/{id}`. These API catalog entries had no description.
> - So an agent that read `managedByPlugin` did not know what it meant
or where a person changes the routine.
> - In chat evals, the agent searched for managed-routine routes and
probed plugin files for over four minutes. It did not reply.
> - This pull request adds one shared description to both catalog
entries. It says the plugin owns the definition, links the board's
plugin settings page, and tells the agent not to rewrite, detach or
duplicate the routine.
> - The benefit is that the agent can answer at once and send the person
to the correct page.
## Linked Issues or Issue Description
No public issue exists. This is a bug report.
**What happened**
A person asked the CEO agent in chat to rewrite a routine that a plugin
manages. The agent read the routine and saw `managedByPlugin`. No tool
result or catalog entry told it what that field means. The agent
searched the API catalog ("plugin managed routine", "managedByPlugin",
"detach plugin managed routine"), tried to read plugin UI files, and
listed agents. After more than four minutes it had not replied, and the
chat timed out. This happened in two separate runs.
**Expected behavior**
The agent replies promptly. It says that the plugin owns the routine,
does not rewrite or copy it, and links the plugin settings page where a
board user changes what the plugin exposes.
**Steps to reproduce**
1. Install a plugin that declares a managed routine (`routines.managed`
capability, top-level `routines[]`).
2. Open a chat with the CEO and select the routine.
3. Ask: "Rewrite this plugin's daily maintenance routine."
4. Look at the agent's `search_api` results for `GET /api/routines/{id}`
and `PATCH /api/routines/{id}`. They have no description.
## What Changed
- `server/src/routes/openapi.ts`: add a shared
`PLUGIN_MANAGED_ROUTINE_GUIDANCE` description to `GET
/api/routines/{id}` and `PATCH /api/routines/{id}`. It names
`managedByPlugin`, links `[plugin
settings](/<prefix>/company/settings/instance/plugins/<pluginId>)`, and
tells agents to reply with that link instead of rewriting, detaching or
duplicating the routine. The `PATCH` entry also states the existing
agent rule (own routines only, no reassignment).
- `server/src/__tests__/plugin-managed-routine-guidance.test.ts`: new
regression test. The queries the agent actually used now find these
entries with the guidance, and the linked settings route is mounted in
`ui/src/App.tsx`.
## Verification
- Red then green: `cd server && npx vitest run
src/__tests__/plugin-managed-routine-guidance.test.ts` failed 6/6 before
the change and passes 6/6 after.
- `cd server && npx vitest run
src/services/native-runtime/runner-api.test.ts
src/__tests__/openapi-routes.test.ts
src/__tests__/person-only-page-guidance.test.ts
src/__tests__/case-history-api-discovery.test.ts
src/__tests__/skill-import-discovery.test.ts` passes.
- `cd server && npx tsc --noEmit -p .` passes.
- Live chat eval `routines.plugin-managed-routine` on a branch with this
commit: see the comment on this PR for the run.
## Risks
- Low risk. The change adds documentation text to two API catalog
entries. Routes and permissions do not change.
- The catalog search scores description words, so these two entries can
rank a little higher for queries about plugins or managed resources.
## Model Used
- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code
execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandClaude Opus 5.5
|
e1010e7678 |
perf(api): per-phase Server-Timing for the Mine inbox snapshot (#20408)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - The Mine inbox page reads one snapshot: `GET
/api/companies/:companyId/inbox/mine`.
> - This read is slow. It takes 280–570 ms on a dev instance and 1.9–3.2
s on staging.
> - The response has only the total `paperclip_api` Server-Timing value.
We cannot see which phase is slow.
> - Issue detail already reports per-phase Server-Timing
(`issue-read-timing.ts`). The Mine snapshot does not.
> - This pull request adds per-phase Server-Timing to the Mine snapshot,
including each lookup inside the issue list.
> - The benefit is that DevTools on any instance shows the slow phase
directly. A follow-up PR uses it to prove a query rewrite.
## Linked Issues or Issue Description
**What existing behavior does this improve?**
The Server-Timing header of `GET /api/companies/:companyId/inbox/mine`.
**Subsystem affected**
Server API: issues routes and the issue list service.
**Current behavior**
The header has only `paperclip_api;dur=…`. Slow access checks, issue
lists, parent context, failed-run reads and list enrichment lookups all
look the same.
**Proposed behavior**
The header also has `paperclip_inbox` (total) and one `inbox_<phase>`
entry for each phase that ran.
**Reason and benefit**
We need phase evidence from staging and production to rank the slow
parts. No telemetry is sent; this is a response header only.
## What Changed
- Added `server/src/services/inbox-mine-timing.ts`. It times closed,
content-free phases and formats a Server-Timing value. It emits no spans
and no telemetry events.
- `issueService.list` accepts an optional third argument, `{ timePhase
}`. It wraps the list's own reads: `query`, `labels`, `active_runs`,
`user_stats`, `read_stats`, `last_activity`, `archives`, `blocked_by`,
`live_descendants`, `attention`. Callers that do not pass it are
unchanged.
- The Mine route reports `inbox_access`, `inbox_issues`,
`inbox_parents`, `inbox_failures`, `inbox_failure_context` and
`inbox_issues_<list phase>`. The existing middleware merges these with
`paperclip_api`.
- The index-projection list no longer awaits a `Promise.all` of
already-empty attention maps. The result is the same.
- `inbox-mine-routes.test.ts` asserts that all phases are present with a
valid duration.
## Verification
- `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/inbox-mine-routes.test.ts` — 6/6 pass, including the new
header assertion.
- `tsc --noEmit -p server` — no errors in the changed files.
- `pnpm verify --explain` selects 2,417 of 3,204 test files for this
change, because `services/issues.ts` is imported almost everywhere. I
did not repeat that full suite locally (AGENTS.md §7); required CI
supplies it.
- Manual: open the Mine inbox, then DevTools → Network → `inbox/mine` →
Timing. The phases show under Server Timing.
## Risks
- Low risk. The header adds about 13 short entries. Phase names are
fixed strings and contain no task data.
- `timePhase` is optional, so other `svc.list` callers do not change.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits). Standard context window; extended thinking
not used.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandClaude Opus 5.5
|
7a93cc6215 |
fix(server): redact invitation tokens from agent run logs (#20360)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - People join a company through an invitation link. The link carries a bearer token in its URL path. > - An agent can create an invitation in chat and give the person the join link. > - The server writes each agent run to a run log file (`.ndjson`). This file is a plain diagnostic copy. Local agents run as the same OS user, so they can read it. > - #20343 removed invitation tokens from HTTP logs. The run log still kept the full join link and the raw token. > - This pull request removes invitation tokens from run-log chunks when the server writes them. > - The benefit is that a live join link does not stay in a file that other local agents can read. The person still gets the working link in the chat reply. ## Linked Issues or Issue Description Refs #20343 (same token policy for HTTP logs). **What happened?** An agent created an invitation in chat and replied with a `/invite/<token>` join link. The agent run log file under `data/run-logs/` kept the full token. The token was in the agent message and in the tool result. **Expected behavior** The run log keeps the route and drops the token: `/invite/***REDACTED***`. A raw `pcp_invite_...` token also becomes `***REDACTED***`. The chat reply that the person reads keeps the working join link. **Steps to reproduce** 1. Check out master. 2. Run `server/src/services/heartbeat/run-log.test.ts` from this PR. 3. The "run-log invitation token privacy" tests fail. The token stays in both the legacy text chunk and the canonical native envelope. **Paperclip version or commit** Master at the merge base of this branch. **Deployment mode** Source server, local run logs (`local_file`). The same functions also format object-store run logs. ## What Changed - Add `redactInvitationTokens()` in `server/src/middleware/invitation-token-redaction.ts`, next to the HTTP log redactor from #20343. It removes the token from `/invite/<token>` segments and from raw `pcp_invite_` tokens inside free text and JSON text. The character classes stop at JSON, Markdown and URL delimiters, so the record structure stays valid. `redact-sensitive.ts` stays byte-identical, because the chat eval pins its exact bytes before it accepts a real invitation link. - Apply it in `compactRunLogChunk()` and `redactCanonicalNativeRunLogChunk()`. These are the two functions that format every agent run-log chunk before the store writes it. The run excerpt columns use the same chunk, so they are also redacted. - `canonicalNativeRunLogRedactionForms()` also yields the earlier, unredacted projection. A run log written by an earlier release still verifies after an upgrade. - `restoreCompactedEvent()` accepts a truncated record that the earlier compactor wrote with the token. Cold recovery of such an old log still works. - Add a test that proves the chat reply comes from database run events, not from the run log. The reply keeps the full join link, and the presentation path never reads the run log. ## Verification - From `server`: `node ../node_modules/vitest/vitest.mjs run --config vitest.config.ts src/services/heartbeat/run-log.test.ts src/__tests__/invitation-token-redaction.test.ts src/__tests__/redact-sensitive.test.ts src/__tests__/heartbeat-run-log.test.ts src/__tests__/logger-invitation-redaction.test.ts src/services/native-runtime/native-run-log-watermark.test.ts src/services/native-runtime/native-run-log-recovery.test.ts src/__tests__/heartbeat-run-output.test.ts src/services/native-runtime/agent-chat-response-recovery.test.ts` passes. - Before the `run-log.ts` change, the two new redaction tests in `run-log.test.ts` failed (red), then passed (green). - `native-run-log-watermark.test.ts`: a legacy-compacted record with a token in its retained head cold-recovers. It failed with `native_log_source_coverage_unproven` before the restore fix. - `agent-chat-response-recovery.test.ts` (embedded Postgres): a final reply with a join link is saved as the exact comment with the token. No run-log read happens. - `tsc --noEmit -p server` shows no errors in the changed files. - A live chat eval (`access.invite-teammate`) run with this change confirms the run-log file has no token. The run is linked in a comment. ## Risks - Low risk. Run-log readers (run transcript view, feedback export, support inspection) now show `***REDACTED***` in place of invitation tokens. This removes a live credential from a diagnostic copy. It does not change the chat reply or the stored run events. - A path segment after `/invite/` in free text is always redacted, even if it is not a token. This loses only diagnostic detail. - A token that a provider streams across two separate chunks is not joined first. Each chunk is redacted on its own. A complete token in one chunk, such as a final message or a tool result, is removed. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
b09f17fbfd |
fix(agents): repair queued setup through task connection cards (#20361)
## Thinking Path > - Paperclip manages AI agents doing work for people. > - Agent verification must finish before an accepted task can start. > - A missing AI account default can prevent verification before a provider starts. > - Lifecycle policy treated that owned configuration refusal as a temporary failure and retried each minute. > - The accepted turn remained Queued without explaining the prerequisite. > - Park account configuration failures and reuse the task's existing inline connection card. > - Successful repair rechecks verification and releases the original queued turn under its original authority. ## Linked Issues or Issue Description **What happened?** An OpenCode agent selected the responsible user's OpenRouter default, but that default did not exist. Verification retried unchanged configuration indefinitely and its accepted task stayed Queued. **Expected behavior:** Preserve accepted work, explain the setup prerequisite, and let the appropriate person repair it using the existing task connection UI. Completion must verify the agent before starting the same queued turn. Genuine pause, approval and access gates remain effective. **Reproduction:** Create an OpenCode Runner agent with a responsible-user OpenRouter binding and no personal default; assign a task while verification is pending. The base retries verification and shows an unexplained queue. Related: #18702 handles pending task connection recovery after account repair. This PR handles the earlier agent-verification boundary. #20330 addresses a separate Codex governed-wait timeout and is already merged. ## What Changed - Share owned account-selection failure evidence through a pure library contract. Lifecycle policy parks those configuration failures with fixed, redacted guidance; temporary infrastructure errors retain their retries. - Batch setup-state reads for unstarted queued turns. Distinguish manual repair from automatic setup retries and preserve Stop, started/history state and lifecycle holds. - Reuse the existing `connection_intent` card, account selection and completion APIs. Its controller-only `agentSetup` marker accepts only never-started legacy turns; runtime tokens still require a running run. - Address the card to the lifecycle verifier's account owner, preserving the task's execution principal and private-task visibility. Attach missing cards on admission, verification, startup and full recovery, including retained retries and continuations. Repeated delivery and publication remain bounded. A new queued turn supersedes the setup card of a stopped turn; generic connection cards retain cross-run reuse. - Recheck lifecycle verification after usable-account completion, then release the existing turn through the normal readiness path. Preserve decline, Stop, reassignment and changed setup-owner gates. Fence an in-flight verification failure from overwriting a repair; add no provider wake or recurring worker. - Park owned account refusals during resume as well as initial verification. Bind setup cards to the lifecycle operation, settle obsolete deliveries, and keep explicit Retry guidance accurate. - Reuse the router’s existing lifecycle pin for setup repair without rerouting the task, advancing the pool cursor, or replacing its saved binding. Legacy direct-account cards remain readable. - Hide the incorrect Resume banner during setup, retaining genuine manual/budget pause controls. Document the execution and retry contract. ## Verification - Red: eight original behavior assertions failed on the base, including the real selector's retry decision and the real batched queue projection. Six owned selection cases failed classification. Temporary-error presentation, different task/verifier users, retained retry/continuation recovery and the setup Resume banner also have failing-before/passing-after regressions. - Green: 430 affected backend, lifecycle, shared-contract and UI tests passed before the final review corrections. Subsequent focused runs cover one inline card, real account selection/completion, duplicate delivery/publication, verification before readiness, unchanged run authority, private tasks, no extra provider wake, decline/Stop/reassignment, company isolation, invalid runtime tokens, transient retries and redacted diagnostics. - Final setup/card and generic connection-service suites: 49 tests passed. The stopped-turn case failed before the correction, then passed with real account completion, the stopped source remaining cancelled, and the replacement source remaining queued. - After rebase, all 129 focused assertions passed. The local queue suite’s 88 execution assertions passed repeatedly, but its database-fixture teardown exceeded both 30-second and 120-second hooks. An isolated queue case completed, and the setup/connection suites fully passed. The original queue test and cleanup remain unchanged; broad CI supplies the full-suite result. - Server/UI typechecks, module boundaries, token gates and diff checks passed. `pnpm verify --explain` requested broad CI because the execution-contract document changed; no redundant full local suite was substituted. - Real source acceptance: macOS arm64, Node 26.4.0, isolated embedded PostgreSQL and local board identity. Created an OpenCode/OpenRouter agent and task through the API with no default. Admission retained one unstarted turn and produced the existing inline Connect OpenRouter card. The browser opened its credential form and closed it without entering credentials. The same card/run persisted after server restart and reload, with the setup label and no incorrect Resume. The browser also reproduced Stop followed by a new queued turn retaining the stale card on the prior head. On `12f609c98372fd1e50644028db75cfcc8deb210d`, startup superseded the stale card and retained the same replacement turn, with the original turn still cancelled and no provider execution. - The earlier CI case-search failure also reproduces on exact base `0e3fdebea8257bdff11573081c76b306f12db230` without this PR. Its repair is now in master via #20371; this branch is rebased onto it and contains no API-search changes. - Staging's actual Runtime test returned “Connect an account and choose your personal default” before provider startup. The original turn later started independently on deployed code, then hit a separate workspace-export failure. Automatic file recovery subsequently marked that run successful; its task remains Blocked and the trial’s useful output is unverified. This PR is not deployed; provider authentication through its card and downstream publication remain unverified. No account, credential, default, permission or binding was changed during diagnosis. - Previous revision review: Greptile 5/5 on `12f609c98372fd1e50644028db75cfcc8deb210d`, with all four inline findings addressed and resolved. [Full CI passed](https://github.com/paperclipai/paperclip/actions/runs/42713494508) on that revision: 59 successful checks, one neutral check and three intentional skips, with no pending or failed checks. - Latest revision `976bcae993805a8a2222395bbff9394f6791b424`: resume, router and Retry-copy regressions failed before repair. Green: 92 lifecycle/setup/shared tests, 83 setup/generic-connection/router tests, and 137 UI/shared tests. Server/UI typechecks, module boundaries and token gates passed. [Full CI passed](https://github.com/paperclipai/paperclip/actions/runs/42720731137) on this exact revision: 59 successful checks and three intentional skips, with no pending or failed checks. Greptile reviewed this head at 5/5; all six review findings are addressed and resolved. Real Stop/retry acceptance retained the same replacement turn and current-operation card through two actual server restarts, with no provider execution. ## Risks - Known account configuration failures now require repair or explicit Retry. Temporary failures keep automatic retries; accepted work and authority remain intact. - The projection adds one company-scoped batch read only for unstarted queued turns. Card attachment examines at most 100 active candidates and adds no timer, worker, schema or migration. - The optional stored setup-card marker requires an aware reader during rollback. No run/session/checkpoint identity format changes. Setup presentation grants no access and cannot bypass verification or other gates. ## Model Used - OpenAI GPT-6 (Codex), with reasoning, tool use, code editing and test execution. The exact serving model identifier and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |