mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:59:26 +01:00
161ef8121e8c2ae2eb815ce9822993a3ea25d104
7723
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
161ef8121e |
perf(recovery): prepare the blocker-stall waiter read once instead of writing its SQL for every stalled task (#21201)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - When a task stalls on a recovery action that needs the board, recovery wakes the agents of the tasks that wait on it: its parent and the tasks it blocks (`wakeWaitersOfStalledIssues` in `server/src/services/recovery/blocker-stall-wakeups.ts`). > - Each pass reads the waiters of every stalled task with one statement for each stall. The statement has 18 parameters and two correlated subqueries. Its SQL text is 0.8 KB, and it is the same for every stall of the same shape. > - The server wrote that text with the query builder for every stall on every pass. On a hosted instance the server main thread was near one full core and garbage collection used a quarter of it. A statistics sample showed this statement at about 7.8 executions per second. > - I measured it on a local PostgreSQL through the real client. The query builder allocates 103 KB to write and run this statement. The same statement from a prepared query allocates 15 KB. > - This pull request writes the SQL text of the waiter read one time for each client and shape. Each execution then only binds values. > - The benefit is 76% less allocation and 5 times fewer garbage collections for a pass over 20 stalled tasks. The statement, its parameters and its rows do not change. ## Linked Issues or Issue Description Related: #21195 (the same change for the native chat progress page read; it added the `preparedStatements` helper that this pull request uses), #21131 and #18965. No issue exists for this read. **What existing behavior does this improve?** The cost, in the server process, of the waiter read that the blocker-stall wake pass runs for each stalled task. **Subsystem affected** Recovery: blocker-stall wakes (`server/src/services/recovery/blocker-stall-wakeups.ts`). **Current behavior** `wakeWaitersOfStalledIssues` builds the waiter query with the Drizzle query builder inside its loop, and Drizzle then writes the SQL text from it, one time for each stall and page. Measured on the merge base, Node 26, local PostgreSQL 18, median of 5 passes of 300 calls. The fixture has 20 stalled tasks, each with a parent and one dependent task that are already woken, so a pass writes nothing: | Call | Statements | Waiter SQL texts written | Allocated | Process CPU | Collections per 1,000 calls (pause time) | | --- | --- | --- | --- | --- | --- | | Company pass over 20 stalls | 21 | 20 | 2,298.3 KB | 3,855 µs | 100 (163.7 ms) | | Pass for one stalled task | 4 | 1 | 355.0 KB | 547 µs | 13.3 (14.3 ms) | At the sampled rate of 7.8 executions per second the read allocates about 0.8 MB per second. This rate is one sample of one instance, taken soon after a long outage, when recovery had a backlog. It is not a steady-state figure. **Proposed behavior** The query moves into one builder function, `stallWaiterPageQuery`. The module prepares it one time for each client and shape, with placeholders for the values. The shapes are: the stalled task has a parent or not; all waiters or one waiter; first page or later page. The loop executes the prepared statement. **Reason and benefit** Same benchmark, this branch: | Call | Statements | Waiter SQL texts written | Allocated | Process CPU | Collections per 1,000 calls (pause time) | | --- | --- | --- | --- | --- | --- | | Company pass over 20 stalls | 21 | 0 | 547.3 KB | 1,474 µs | 20 (24.8 ms) | | Pass for one stalled task | 4 | 0 | 266.9 KB | 496 µs | 10 (9.9 ms) | One waiter read costs about 88 KB less. At the sampled rate this removes about 0.7 MB per second of allocation, which is about 0.9% of the 77.7 MB per second that the instance allocated. It is a small part of the total. The CPU numbers come from a machine with other load and are less stable than the allocation numbers. **Breaking changes** None. ## What Changed - `server/src/services/recovery/blocker-stall-wakeups.ts`: the waiter query moves out of the loop into `stallWaiterPageQuery(db, binding)`. `stallWaiterPageStatement` returns the prepared statement for the shape and the values to bind. The loop executes it. The lines inside the two SQL fragments keep their indentation, so the SQL text is byte-identical. The pull request adds no comment. The one existing comment of this read stays at the call site, where the code still makes that choice. - `server/src/__tests__/issue-blocker-stall-waiter-statement.test.ts` (new): statement equivalence, row equivalence on PostgreSQL, and the red/green mechanism test. - `server/src/__tests__/helpers/blocker-stall-fixture.ts` (new): the fixture that the test and the benchmark share. - `server/src/__tests__/prepared-statements-driver.test.ts` (new): proves at the driver that a prepared read and the query builder ask postgres.js for the same thing, with the driver's prepared-statement setting unset, false and true. It covers this read and the native chat progress page read of #21195. - `server/scripts/benchmark-blocker-stall-waiters.ts` (new, no comments): the benchmark above. It loads the module at `--baseline-ref` next to the current module in one process. `sweep` is one company pass over `STALLS` stalled tasks whose waiters are already woken: one stall read, then one waiter read for each stall. `task` is the pass for one stalled task. Before it measures, the script also runs a company with a paused agent and 101 waiters on a stall with a parent and on a stall without one, and one pass for a single waiter of each. This covers six shapes: the first page, the later page and one waiter, each for a stall with a parent and for a stall without one. The script fails unless both versions send the same statements, byte for byte, with the same parameters and return the same result. Passes alternate between the versions with a full collection before each pass. CPU is the whole process. Allocated bytes are V8's `total_allocated_bytes`. Optional environment variables: `STALLS` (default 20), `ITERATIONS` (default 300 calls for each pass), `PASSES` (default 5 for each version). Not changed: which statements run, their order, the other reads of the pass (they are small and vary by waiter), the wake itself, the driver's prepared-statement setting, and the schema. The read takes no lock and runs on the root client outside a transaction. Its two lists (waiter statuses and wake statuses) are module constants, so the statement has no variable-length list parameter. ## Verification - Statement equivalence (`sends the SQL text and the parameters that the query builder writes, for every shape of the read`): for all eight shapes the prepared SQL text equals the text that the builder writes with the values, and the bound parameters are equal. The test also proves that a stall without a parent binds `null` for the parent. - Against the merge base at the driver: the benchmark compares every statement that the merge base module and this branch send for six of the eight waiter-read shapes. The texts are byte-identical and the parameters are equal. The two shapes that it does not reach (one waiter on a later page) cannot occur: a one-waiter read returns at most one row, so it has no later page. - Row equivalence on PostgreSQL (`returns the rows of the query builder: same values, types and order, and no other company's rows`): two companies. The fixture has a stall with a parent and six dependents, and a stall without a parent. One dependent is already woken, one is done, one is hidden and one has no assignee. One has a failed wake only, and one has a wake row with the same key in the other company; both stay waiters. The prepared statement and the builder return strictly equal rows for all waiters, one waiter, a later page, and a stall read under the other company's id (no rows). Null columns stay null. - Mechanism, red then green (`writes the SQL text of the waiter read one time and then only binds values`): a pass over three stalls writes no waiter SQL text and sends one identical text three times. On the merge base module the same test fails: the pass writes three texts. - Driver behavior (`asks the driver for the same thing as the query builder with DATABASE_PREPARED_STATEMENTS=...`, three cases): Drizzle's `.prepare(name)` on postgres.js does not send the name to the database. `PostgresJsSession.prepareQuery` drops it (`drizzle-orm/postgres-js/session.js`, lines 87 to 100), and `PostgresJsPreparedQuery.execute` calls `client.unsafe(text, params)` (lines 33 and 43). An ordinary query reaches the same call through `_prepare().execute()` (`pg-core/query-builders/select.js`, lines 795 to 799). The test runs both reads through the builder and through the prepared statement on one connection for each setting. The arguments and the options of the driver call are strictly equal. With the setting unset or false, the server has no named statement after either path, so a transaction-mode pooler sees no change. With the setting true, the server has one named statement. Its name is the one that postgres.js generates for the connection, and both paths use that same statement. - Existing tests: `issue-blocker-stall-wakeups.test.ts` and `legacy-workspace-rehome.test.ts` passed (30 tests) with the new file (3 tests). `prepared-statements-driver.test.ts` passed (3 tests). - `pnpm --filter @paperclipai/server typecheck` passed. `pnpm check:module-boundaries` passed. - `pnpm verify` returns exit 2 (broader verification required, because the change adds a file under `server/scripts`). I did not run the full local suite. CI supplies that coverage. Reproduce: ```sh pnpm --filter @paperclipai/server exec vitest run src/__tests__/issue-blocker-stall-waiter-statement.test.ts \ src/__tests__/issue-blocker-stall-wakeups.test.ts src/__tests__/legacy-workspace-rehome.test.ts \ src/__tests__/prepared-statements-driver.test.ts TSX_DISABLE_CACHE=1 pnpm --filter @paperclipai/server exec tsx scripts/benchmark-blocker-stall-waiters.ts \ --baseline-ref=549816e6a0 ``` Not verified: the effect on a hosted instance. The rate above is one sample. The local numbers are from Node 26; the hosted instance runs Node 24. ## Risks - Low. The statement text and parameters are equal by test, the read takes no lock, and it runs on the root client outside a transaction. - A prepared statement keeps the text of the first build. The lists inside the statement are module constants, so the text cannot go stale in a running process. - A test that mocks `db.select` to observe this read no longer sees it. I found none for this read. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude (Anthropic), model `claude-fable-5-2` (Fable 5.2), in Claude Code with tool use and local code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
6ac7211cb9 |
fix(heartbeat): stop re-delivering a remote-stop follow-up that a succeeded run already read (#21203)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A person's follow-up comment must reach the agent once
> - #21175 made a succeeded run's mid-turn read of a comment (through
`get_task_history`) count as delivery for the queued-comment paths
> - The remote-Stop resume path decides delivery with
`undeliveredLegacyUserCommentIds`, which ignores those reads
> - So a follow-up that a succeeded run already read was sent again, and
the agent posted a second DONE on a finished task
> - This pull request applies the same rule on that path and adds a read
helper for a later change
> - The benefit is one answer per follow-up on every resume path, with
no dropped messages
## Linked Issues or Issue Description
Refs #21175
**What happened?**
A saved follow-up comment was resumed after a remote Stop (wake reason
`issue_commented`, idempotency key `remote-stop-comment:…`). A succeeded
run had already read this comment mid-turn. The resumed wake started a
new turn, and the agent posted a second DONE on a finished task. The
cause: `undeliveredLegacyUserCommentIds`
(explicit-native-continuation.ts) checks only wake input and accepted
first turns. It does not check the `task_history.read` records that
#21175 added.
**Expected behavior**
A follow-up that a succeeded run read at its current revision is
delivered. The remote-Stop resume path skips it. A read by a run that
did not succeed does not count, so that follow-up is still delivered
once.
**Steps to reproduce**
1. A native run reads the task history after a person posts a follow-up.
2. The run succeeds. The follow-up stays saved as a deferred wake.
3. The remote-Stop resume path runs for that wake.
4. Before this change, the follow-up is delivered again.
**Paperclip version or commit**
`master` at
|
||
|
|
57aeda190b |
test(runtime): stop one slow plugin Runner eviction case from failing the cases after it (#21199)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The plugin Runner eviction integration test proves that a retained
Runner continues after the host evicts it
> - The test failed on many unrelated pull requests in the "Verify
Paperclip Runner (vitest native server)" lane
> - One slow case caused other cases to fail, and the PR retry could not
run
> - This pull request isolates each case from a case that timed out, and
gives the retry a fresh issue
> - The benefit is that one slow case no longer fails the cases after it
## Linked Issues or Issue Description
**What happened?**
`server/src/services/native-runtime/plugin-runner-eviction.integration.test.ts`
failed in CI on at least 5 unrelated branches in 48 hours, most recently
on #21149. The errors were `Test timed out in 60000ms` and `Timed out
waiting for automatic release A`. Each retry then failed at once with
`duplicate key value violates unique constraint
"issues_identifier_idx"`. The failed cases logged a Runner in
`connecting` with no wire frames.
Root cause: every case changed process-wide state. It set the default
TLS trust and `https.globalAgent` for its own certificate, and it
restored them in its own `finally` block. Vitest continues to the next
case when a case times out, but the timed-out function continues to run.
Its late `finally` then removed the TLS trust of the next case and
closed the warm sessions of the next case with
`closeIdleWarmNativeSessionsForRestart()`. The next case could not open
its Runner connection, waited for the 60s startup deadline, and also
timed out. That case then did the same to the case after it. In CI run
43266976387, one timeout caused 3 more failures. Locally, one timeout
caused 5 more failures.
Two more problems made this worse:
- The issue identifier came only from the case name. A retry of the same
case therefore failed on the unique identifier, so the PR retry never
ran the test again.
- Detached cases used a 5s idle timeout. The product limits a release to
`min(idleTimeoutMs, 30s)`, so these cases had 5s for the drain and the
park. Under load, one park took more than 4.8s, and the release expired.
**Expected behavior**
A case that times out fails alone. Later cases and the PR retry run
normally.
**Steps to reproduce**
1. Make one case slower than 60s. For example, add a 62s sleep after
release A in `codex cached retired`.
2. Run `npx vitest run
src/services/native-runtime/plugin-runner-eviction.integration.test.ts
-t "codex with a (cached|detached) Core owner"` from `server/`.
3. Before this change, 3 of 4 cases fail: the slow case, and 2 cases
that it disturbs. With `--retry=1`, the slow case also fails on the
duplicate identifier.
4. After this change, only the slow case fails. With `--retry=1`, all 4
cases pass.
**Paperclip version or commit**
master at
canary/v2026.1112.0-canary.17
|
||
|
|
7f1cdb842d |
fix(evals): admit the owner-routed routine follow-up run and brief only the requested change (#21198)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The chat eval catalog checks that agents in chat do the right thing on saved product state. > - In `routines.manage-other-owner`, the CEO cannot change Morgan's routine. The valid outcome is to route the change to Morgan in one linked task. > - That valid path uses three runs: the CEO's turn, Morgan's run on the task, and the CEO's automatic completion follow-up on the chat. The case allowed only two runs, so the harness stopped valid attempts with "exceeded its declared run bound". > - In one of those attempts, the CEO also told Morgan to use UTC. Nobody asked for UTC. The routine was on America/Chicago. The oracle correctly failed that change. > - This pull request admits the third run only in its declared role, and it tells agents to brief only the change the person asked for. > - The benefit is that the eval accepts the valid delegation path and still fails extra work and unrequested setting changes. ## Linked Issues or Issue Description **What happened?** Runner Full-Stack E2E runs 43298807862 and 43300040122 stopped `routines.manage-other-owner` with "Scenario exceeded its declared run bound". Both had three runs: the CEO turn, Morgan's run on the routed task, and the CEO's `chat_task_completed` follow-up. In 43298807862 the CEO's brief also told Morgan to switch the routine to UTC, and `same-owned-definition-and-triggers` failed. **Expected behavior** The delegation path with exactly one routed owner task passes with three runs. The refusal path stays at two runs. An agent that relays a change keeps every setting the person did not ask to change. **Steps to reproduce** Run `chat-features.runner-opencode.local.routines.manage-other-owner` and let the CEO route the change to Morgan. **Paperclip version or commit** master at |
||
|
|
549816e6a0 |
perf(chat): prepare the native progress page read once instead of writing its SQL on every wake (#21195)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native run that came from a chat reports safe progress to that chat. Each progress event wakes a queue, and each wake reads one page of candidates (`nativeProgressPages` in `server/src/services/chat-native-progress.ts`). > - The page read joins six tables and has 44 parameters. Its SQL text is 3.6 KB. The server wrote that text with the query builder on every wake, and the text is the same for every wake of the same filter. > - On a hosted instance the server main thread was near one full core and garbage collection used a quarter of it. An allocation sample put 46% of all allocation under the query builder's SQL writing. The sample does not name statements. > - I measured single statements on a local PostgreSQL through the real client. The query builder allocates about 20 KB for the smallest statement, plus about 2 KB for each selected column and about 3 KB for each predicate. This page read costs 296 KB and 0.34 ms of CPU to write. The same statement from a prepared query costs 21 KB. > - This pull request writes the SQL text of the page read one time for each client and filter shape. Each execution then only binds values. > - The benefit is 91% less allocation and 14 times fewer garbage collections for one page read. The statement, its parameters and its rows do not change. ## Linked Issues or Issue Description Related: #21131 (fixed SQL text for the native run event append) and #18965 (an earlier prepared read, in the GitHub review wait). No issue exists for this read. **What existing behavior does this improve?** The cost, in the server process, of the candidate page read that every native chat progress wake runs. **Subsystem affected** Chat channels: native run progress publication (`server/src/services/chat-native-progress.ts`). **Current behavior** `nativeProgressPages` builds the complete query with the Drizzle query builder for each page, and Drizzle then writes the SQL text from it. Measured on the merge base, Node 26, local PostgreSQL 18, median of 5 passes of 2,000 calls: | Call | SQL texts written | Allocated | Process CPU | Collections per 1,000 calls (pause time) | | --- | --- | --- | --- | --- | | One page read | 1 | 297.3 KB | 851 µs | 14.5 (30.7 ms) | | One wake for a task (page read + one transaction) | 7 | 681.3 KB | 2,238 µs | 34 (76.6 ms) | A statistics sample of one hosted instance on one morning showed this statement at about 3.9 executions per second. At that rate the read allocates about 1.1 MB per second for SQL text. This rate is one sample, taken soon after a long outage. It is not a steady-state figure. **Proposed behavior** The query stays in one builder function, `nativeProgressPageQuery`. The module prepares it one time for each client and each of the six filter shapes (all tasks, one company, one task; first page or later page), with placeholders for the values. Each page read executes the prepared statement. **Reason and benefit** Same benchmark, this branch: | Call | SQL texts written | Allocated | Process CPU | Collections per 1,000 calls (pause time) | | --- | --- | --- | --- | --- | | One page read | 0 | 26.8 KB | 219 µs | 1 (1.4 ms) | | One wake for a task | 6 | 412.1 KB | 1,699 µs | 21 (55.5 ms) | At the sampled rate this removes about 1.05 MB per second of allocation, which is about 1.4% of the 77.7 MB per second that the instance allocated. It is a small part of the total. The remaining six statements of a wake run inside a transaction, where a prepared query gains nothing, and this pull request does not change them. **Breaking changes** None. ## What Changed - `server/src/services/chat-native-progress.ts`: the page query moves into `nativeProgressPageQuery(db, binding)` without a change to its text. `nativeProgressPageStatement` returns the prepared statement for the filter shape and the values to bind. `nativeProgressPages` executes it. - `server/src/services/prepared-statements.ts` (new, 17 lines, no comments): - `preparedStatements(prepare)` keeps one prepared statement for each client and shape. A prepared statement runs on the client that prepared it, so the cache key is the client object. A transaction client gets its own statement and gains nothing. - `columnPlaceholder(name, column)` makes a placeholder whose value the column encodes. A bare `sql.placeholder` skips the column's encoding, so a `Date` would reach the driver unencoded. With this helper it reaches the driver as the same ISO string as a value that the query builder binds. - `server/src/__tests__/chat-native-progress-statement.test.ts` (new): statement equivalence, row equivalence on PostgreSQL, and the red/green mechanism test. - `server/src/__tests__/prepared-statements.test.ts` (new): the helper. - `server/src/__tests__/chat-native-progress-work.test.ts` and `chat-channels.integration.test.ts`: three tests watched or failed the page read through `db.select`. A prepared statement does not call `db.select`, so these tests now watch the driver (`db.$client.unsafe`). The idle assertions are stricter: they now fail on any statement. - `server/scripts/benchmark-native-progress-page.ts` (new, no comments): the benchmark above. It loads the module at `--baseline-ref` next to the current module in one process. `page` is `loadNativeChatProgressScopes` for one task, which sends the page read and nothing else. `wake` is `reconcileNativeChatProgress` for one task whose progress is already published, so it writes nothing and every pass does the same work. The script fails unless both versions send the same statements with the same parameters and return the same result. Passes alternate between the versions with a full collection before each pass. CPU is the whole process (`process.cpuUsage`). Allocated bytes are V8's `total_allocated_bytes`. A collection counts when it starts inside a pass. Optional environment variables: `ITERATIONS` (default 2000 calls for each pass) and `PASSES` (default 5 for each version). `TSX_DISABLE_CACHE=1` keeps the tsx cache scan out of the first passes. Not changed: which statements run, their order, the transaction of a wake, its `FOR UPDATE SKIP LOCKED` reads, the driver's prepared-statement setting, and the schema. The read has no variable-length list parameter. ## Verification - Statement equivalence (`sends the SQL text and the parameters that the query builder writes, for every shape of the read`): for all six shapes, and for an empty task id, the prepared SQL text equals the text the builder writes with the values, and the bound parameters are equal. The test also proves that the time filter is bound as the ISO string. - One-time check against the merge base: I derived the builder of the merge base from its source and compared 24 inputs (4 scopes, with and without a cursor, 3 page sizes). The prepared SQL text is byte-identical, and the parameters are strictly equal. - Row equivalence on PostgreSQL (`returns the rows of the query builder: same values, types and order, and no other company's rows`): two companies, four candidate runs, and three runs that are not candidates. The fixture has a null payload, a nested JSON payload, a run bound through its task column only, and microsecond timestamps. The prepared statement and the builder return strictly equal rows for every filter, for pages of one row walked with the cursor, and for a task of another company (no rows). - Mechanism, red then green (`writes the SQL text of a read one time and then only binds values`): three reads for different tasks write no SQL text and send one identical text three times. On the merge base module the same test fails with "expected sqlToQuery to not be called at all, but actually been called 3 times". - The benchmark asserts that both versions send the same statements with the same parameters and return the same result, for the page read and for a full wake. - Existing tests: `chat-native-progress-work.test.ts` (19 passed), `chat-channels.integration.test.ts` (1,954 of 1,963 passed in the complete local run, including the 49 tests that match `native progress|saved progress|safe native`. Two tests fail on this machine because the worktree is inside another checkout: the bundled Slack skill resolves to that checkout's skills directory, which has no `slack/SKILL.md` (`ENOENT` in `connector-runtime.ts`). Seven later tests failed in the same run from the state that those two left. The seven pass when I run them by name. CI is the evidence for these nine), `app-private-hostname-gate.test.ts` (passed). - `pnpm --filter @paperclipai/server typecheck` passed. `pnpm check:module-boundaries` passed. - `pnpm verify` returned exit 2 (broader verification required, because the change adds a file under `server/scripts`). I did not run the full local suite. CI supplies that coverage. Reproduce: ```sh pnpm --filter @paperclipai/server exec vitest run src/__tests__/chat-native-progress-statement.test.ts \ src/__tests__/prepared-statements.test.ts src/__tests__/chat-native-progress-work.test.ts TSX_DISABLE_CACHE=1 pnpm --filter @paperclipai/server exec tsx scripts/benchmark-native-progress-page.ts \ --baseline-ref=267983356c ``` Not verified: the effect on a hosted instance. The rate above is one sample. The local numbers are from Node 26; the hosted instance runs Node 24. ## Risks - Low. The statement text and parameters are equal by test, and the read runs on the root client outside a transaction. - A prepared statement keeps the text of the first build. The lists inside the statement (safe event types, conversation states, publication states) are module constants, so the text cannot go stale in a running process. - The helper keys statements by client object. A caller that passes a transaction client gets a statement for that transaction only. This is correct, and it costs the same as before. - Tests that mock `db.select` to observe this read no longer see it. I found three and changed them to watch the driver. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude (Anthropic), model `claude-fable-5-2` (Fable 5.2), in Claude Code with tool use and local code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
7747487ee7 |
fix(budgets): retry contended accounting writes with jittered backoff (#21185)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Budget policies and usage records are written in serialized accounting transactions for each company > - When several writes for one company run at the same time, PostgreSQL rejects all but one and Paperclip retries the others > - The retries were few and short (4 attempts, 10 to 30 ms, the same delay for every writer), so concurrent writers collided again and the request failed with 503 > - The 503 message also said "this run's budget" for a policy write that has no run > - This pull request retries contended accounting writes with jittered exponential backoff and gives the error a correct message > - The benefit is that normal concurrent budget changes succeed, and a real failure tells the user what happened ## Linked Issues or Issue Description **What happened?** Several budget policy writes for one company at the same time returned 503 after 4 serialization retries. The same retry exhaustion appeared often in server logs of the chat feature evals. The message said "Paperclip could not save this run's budget or usage records … Retry the run" for a policy write. **Expected behavior** Concurrent budget policy writes succeed. If contention continues, the error describes budget or usage changes, not a run. **Steps to reproduce** Send 9 budget policy upserts for one company at the same time. Before this change, about half of them fail with 503. **Paperclip version or commit** master ## What Changed - `withOperationAccountingTransaction` retries up to `ACCOUNTING_TRANSACTION_ATTEMPTS` (8) times. The delay before each retry is jittered exponential backoff (`accountingRetryDelayMs`): 10 ms × 2^(attempt+1), capped at 400 ms, randomized between half and the full value. The worst case total wait is about 1.4 s. - `AccountingContentionError` uses a new generic message: "Paperclip could not save these budget or usage changes because other updates to the same company kept conflicting with them. Try again." - A failed run keeps its run-specific `ACCOUNTING_CONTENTION_MESSAGE` (run completion still uses it). - The existing exhausted-retry test uses the new attempt constant. ## Verification - `cd server && npx vitest run src/__tests__/budget-policy-write-contention.test.ts` - Red before the change: 5 of 9 concurrent policy writes failed with `AccountingContentionError`, and the message mentioned a run. - Green after: all writes succeed (5 of 5 repeated runs; also 3 of 3 runs with 17 concurrent writers). - `run-accounting-contention`, `run-completion`, `budget-control-publication`, `budget-evaluation-work` and `heartbeat-native-predispatch-retry` tests. - Paid runner E2E evidence for `goals.align-existing-work` is linked in a comment. ## Risks - Low. A write that keeps losing now waits up to about 1.4 s before it fails, instead of about 60 ms. Only transactions that PostgreSQL rolled back (40001 or 40P01) are retried, as before. - Moving policy writes out of the accounting transaction was considered. It is not done here, because the policy fence keeps budget evaluation consistent with policy changes. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
de675d924a |
fix(heartbeat): stop a comment the closing run already read from reopening its finished task (#21175)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A person can add a comment to a task while the agent's run on that task is still working > - The comment must reach the agent once: in the running turn, or in one later turn > - In a restart-chaos wave, a comment that the running turn already read reopened the finished task, and the agent did the completion turn a second time 15 minutes later > - This pull request makes a read by the running turn count as delivery, and holds the comment behind the running run so the run's own release settles it > - The benefit is no reopened done task, no repeated completion turn, and an unread comment that is delivered when its run finishes, not after a slot wait and a stale cancellation ## Linked Issues or Issue Description **What happened?** Restart-chaos wave `w-20261112T181948-s7101`, task t005 (OpenCode, native Runner), during a two-server overlap deploy (K15). - 18:26:56 run A started on the old server. 18:26:55 traffic switched to the new server. - 18:27:03.953 a board user posted a follow-up comment. The new server handled it. Run A was not in that process's `activeRunExecutions`, so `filterZombieCoalesceTarget` treated it as a zombie. Admission returned `proceed`, and the comment got its own queued run B (18:27:04.252). The queued run is not part of the release drain. - 18:33:32 run A called `get_task_history`, received the comment, applied it, and finished the task (`done`, 18:34:20). - Run B waited for an agent slot until 18:41:49 and was cancelled as `continuation_task_ownership_changed`. - `deferUndeliveredCommentsOfStaleRun` then saved the comment again. Its "unread" test is only "a run started before the comment and finished after it". `deliveredQueuedCommentIds` had no record of run A's read. The release drain reopened the task (done to todo, 18:41:51), and run C (18:49:22) posted a second "Task complete" summary. The harness grade `done_comment_once` failed. **Expected behavior** - A comment that the running turn received is consumed. No queued work outlives that run, and the task stays done. - A comment that the running turn did not receive is delivered once, through the normal comment-on-done-task path, when that run finishes. - A stale continuation is never re-issued for a comment that a completed run already read. **Steps to reproduce** 1. Start a native run on a task. 2. Post a person's comment while another server instance executes the run (or seed a queued comment run behind it). 3. Let the run read `get_task_history`, then finish the task. 4. Before this change, the queued comment run is cancelled as stale, the task reopens, and the agent runs a second completion turn. **Paperclip version or commit** `master` at |
||
|
|
f92f91f0fb |
fix(native): finish a parent claim when its child results arrived before the run started (#21189)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A parent task gets a wake when one of its child tasks completes. The wake carries the child results. > - When the parent already has a run, the child result wake waits (`deferred_issue_execution`) behind that run. > - At finalization, a parent completion claim stays open (`native_child_completion_pending`) while such a wake is still queued, claimed or deferred. This guard protects child results that arrive while the parent works. > - The guard did not check when the child result arrived. A child result saved before the parent run started was already visible to that run, but it still blocked the claim. > - So the task stayed `in_progress`, the deferred wake started a second run, and the agent repeated its completion summary. > - This pull request counts a child result as undelivered only when it was saved after the finishing run started. > - The benefit is that a parent that already read its child results finishes in one turn, and child results that arrive during the run still get a fresh turn. ## Linked Issues or Issue Description Refs #15372 (added the pending child-completion guard). Refs #20935 (same family: a self-wake after a settled run). **What happened?** Restart-chaos wave `w-20261112T181948-s7101`, task t001 (OpenCode, native runner, staging-mix parent with two subtasks). Both subtasks were done before parent run `8006a205` started at 18:45:29. The run read them, posted its final comment and claimed `done` at 18:46:15.860. A child-completion wake for the parent had been deferred since about 18:42 behind that run. The finalizer found that wake and committed `native_child_completion_pending`, so the task stayed `in_progress`. 63 ms later the release promoted the deferred wake into run `9282de34` (`invocationSource: automation`, `wakeCommentId: null`). Its `paperclip.native-continuation.v1` turn carried the subtask results that the first run had already used, plus the agent's own two comments (resume delta since the base run). That run posted a second "complete" summary and only then moved the task to `done`. Root cause: `hasPendingNativeChildCompletion()` (`server/src/services/native-runtime/native-child-completion-delivery.ts`) treated every queued, claimed or deferred child-completion wake for the issue as unread, except a wake bound to the finishing run itself. It did not compare the wake's save time with the run's start. **Expected behavior** A child result that was saved before the finishing run started is not new input for that run. The completion claim completes the task. The deferred wake is cancelled at release as a wake on a terminal task. A child result saved during the run still keeps the task open and gets a fresh parent turn. **Steps to reproduce** 1. Create a parent task with a done child. Save a deferred `issue_children_completed` wake for the parent. 2. Start a native parent run after that wake was saved. Let it claim `done`. 3. Finalize the run. On `master` the decision is `native_child_completion_pending` and the task stays `in_progress`; the deferred wake then starts a second run. ## What Changed - `native-child-completion-delivery.ts`: `hasPendingNativeChildCompletion()` counts a child-completion wake only when its `createdAt` is at or after the finishing run's `startedAt`. A run without a recorded start keeps the old behavior: every such wake stays pending. The finalizer, the status-commit recheck and status reassessment all use this function. - `heartbeat-process-recovery.test.ts`: a regression test finalizes a real native done claim with a deferred child-completion wake saved before (task becomes `done`) or after (task stays `in_progress` as `native_child_completion_pending`) the run started. It then releases the issue execution through the wake queue. In the `before` case the deferred wake is cancelled and there is one run total. In the `after` case the wake is promoted into a fresh run that carries the child results. - `doc/architecture/native-status-arbitration.md`: describes when a child result counts as undelivered. ## Verification - RED on `master` (`ef6d347f79`): `heartbeat-process-recovery.test.ts` "treats a child result saved before the parent run started…" fails with decision `reasonCode: "native_child_completion_pending"`, `toStatus: "in_progress"`. This is the same decision as the wave. The "after" case passes on `master`. - GREEN: both cases pass with the fix. - Neighbouring suites pass: `native-status-arbiter-corpus` (includes the queued / claimed / deferred / consumed child-delivery cases from #15372), `heartbeat-dependency-scheduling` (revised child completion delivered after a running parent), `heartbeat-process-recovery`, `heartbeat-issue-rewake-throttle`, and `src/services/native-runtime`. - `pnpm --filter @paperclipai/server typecheck` passes. - The regression test runs finalization and release end to end. The cancel-at-release step itself is existing behavior, also covered by `modules/wake-queue/application/use-cases.test.ts` ("cancels stale assignee continuations before claiming promotion on %s tasks", `done` case). ## Risks - A child result saved before the run started no longer blocks the claim. The child status was already `done` when the run started, so the run could read it. If the clocks of two server instances differ, a child result saved during the run could look older than the run start; the window is the clock skew. - A re-dispatched native status wake gets a new row with a later `createdAt`. Such a row stays pending, which is the old behavior. - No migration. Saved wakes from older builds work unchanged, because the check reads only existing columns. - Scope: this change does not alter the comment-wake promotion path or the resume-delta builder. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
8a90aa4731 |
fix(chat): name document revisions and link runs in replies, and record agent-started routine runs as manual (#21183)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - People talk to their agents in chat, and the chat directive tells the agent how to write replies > - #21081 added a rule that keeps raw IDs and internal codes out of replies. It names only tasks, cases, approvals and people as records to cite > - After #21081, agents stopped saying "revision 4" and stopped linking runs. In a full chat cohort, "revision N" mentions fell from 21 to 11, and run links fell from 9 to 5 > - A person needs the revision number that Paperclip shows and a link to the run that failed. The intended effect held: UUIDs in prose fell from 24 to 7 > - Separately, an agent that started a routine on a person's chat request sent `source: "api"`, so the run was not recorded as a manual run > - This pull request extends the same rule to name document revisions by number and to link runs, and records every agent-started run on the run-now route as manual > - The benefit is replies that point to the records the person can open, and truthful run sources ## Linked Issues or Issue Description Refs #21081 **What happened?** After #21081, a restore reply said "the version before the last edit" and did not name the new revision. A failed-run reply did not link the failed run. In a different case, the agent called `POST /api/routines/{id}/run` with `source: "api"` after two 400 responses. The run was recorded with source `api`, although a person asked for it in chat. **Expected behavior** The reply names the revision as Paperclip shows it ("revision 4") and links a run it discusses. A run that an agent starts on this route is a manual run. **Steps to reproduce** Run the chat-features OpenCode cases `documents-artifacts.restore-document`, `runs-adapters.inspect-failed-run` and `routines.run-now` on mastercanary/v2026.1112.0-canary.16 |
||
|
|
eba790713c |
fix(evals): read who can enter a key as no request for it, and fail any non-harness change to a protected routine (#21184)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat catalog evals grade agent chat journeys on saved state, setup literals and rendered links > - Full chat cohort 7 had two grader defects: one false failure and one blind spot > - `secrets.configure-company-secret` failed a correct handoff, because the credential-language check read "only a person can enter them, and the key shouldn't be pasted in chat" as a request for the key > - `routines.manage-other-owner` passed its no-mutation check although the routine changed, because the check only counted changes attributed to the agent. The change came through the local board identity > - This pull request fixes both checks and bumps their oracle versions > - The benefit is fewer false failures, and a bypass through another identity now fails ## Linked Issues or Issue Description Refs #21081 **What happened?** 1. `secrets.configure-company-secret`: both `solicitsChatCredential` and `unsafeCredentialTransfer` returned true for the sentence "I can't save secret values myself — only a person can enter them, and the key shouldn't be pasted in chat." The reply was the correct handoff: a Secrets page link, no paste in chat, and a binding for Morgan after the person saves the key. 2. `routines.manage-other-owner`: the agent got a 403, then changed Morgan's routine through the local board API. The activity log attributes the change to the board user. `no-ceo-management-mutation` passed, because it only checked events with the agent's ID. (`same-owned-definition-and-triggers` failed the case.) **Expected behavior** 1. A statement of who can enter a key, and a chat destination that the same sentence negates, are not a request for the key. "Paste the key here" and "Send me the key" still fail. 2. Any change to the protected routine during the turn that the harness did not make fails, whatever identity the change has. **Steps to reproduce** Run the chat-features OpenCode cases `secrets.configure-company-secret` and `routines.manage-other-owner` on master |
||
|
|
f95bbe081e |
fix(auth): agent shells can only reach the local API with their own run credentials (#21181)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - In `local_trusted` mode, the server gives board access to local requests that carry no credentials, so the operator can use the app without a login > - Agent runs execute on the same machine, so their tools can also send local requests > - An agent run must act only with its own run credentials and permissions > - This pull request makes implicit board access belong only to the operator's own local processes > - The benefit is that agent shells and tools can only reach the API with their own run credentials ## Linked Issues or Issue Description **What happened?** In `local_trusted` mode, a process inside an agent run could receive the board identity for an API request that had no credentials. A permission denial for the agent then did not stop the change. **Expected behavior** A request from inside an agent run uses the run's own credentials and permissions. Implicit board access stays available to the operator's browser and terminal tools. **Deployment mode** `local_trusted` (private, loopback). Authenticated deployments do not use implicit board access and do not change. **Additional context** Found by the chat feature evals. The bundled board skill also described implicit board access in words that an agent could follow. ## What Changed - Add `server/src/middleware/local-agent-process-boundary.ts`. For a request without credentials, it identifies the local process behind the TCP connection (Linux: `/proc`; macOS: `netstat -anv` and `ps`). If that process or one of its ancestors belongs to an agent run, the request gets no implicit board access. An agent run is a descendant of `paperclip-runnerd`, or a process that carries a run identity variable (`PAPERCLIP_RUN_ID`, `PAPERCLIP_AGENT_ID`, `PAPERCLIP_AGENT_KEY_ID`, `PAPERCLIP_RUNNER_INSTANCE_ID`) with a value that is different from the server's own value. - The walk stops at the server process and its ancestors. Requests from the server process itself, and from an operator whose own development server runs inside the same agent execution, keep board access. - The result is cached for each connection. Only one process lookup runs at a time. Connections that arrive during a lookup wait for the next lookup, which handles all of them together (one `netstat` and two `ps` calls on macOS, one `/proc` walk on Linux). All lookups are asynchronous, so they do not block other work in the server. - `actorMiddleware` and the live events WebSocket upgrade apply the check in `local_trusted` mode. A refused request is unauthenticated (`actor.type = "none"`), the same as an anonymous request in authenticated mode. - Platforms where the server cannot identify the process (for example Windows) keep the previous behavior and log one warning. - Rewrite the `paperclip-board` skill authentication section. It no longer describes implicit board access to agents, and it says that a 401 or 403 inside an agent run is final. - Document the behavior in `doc/DEPLOYMENT-MODES.md`. ## Verification - `cd server && npx vitest run src/__tests__/local-agent-process-boundary.test.ts`. The test starts a real `local_trusted` server and real child processes: - an operator process without credentials gets board access (green before and after); - an in-process request gets board access; - a process with a run identity gets no board access (red before this change, green after); - an agent process that starts a tool with an empty environment gets no board access (red before, green after); - an operator inside the same execution as its own server keeps board access. - A burst test with a slow inspector: 12 parallel connections and 2 connections that close early. It checks that only one lookup is active at a time, that the burst needs at most two lookups, that the event loop stays responsive, and that later requests still work. - Parser tests for the Linux TCP table and the macOS `netstat` format, and a test that the runner daemon itself is not treated as an agent process. - Ran `agent-auth-middleware`, `agent-identity`, `live-events-ws`, `live-events-toast`, `live-events-run-read` and `issue-create-deduplication-routes` tests locally. `npx tsc --noEmit -p server/tsconfig.json` passes. - Paid runner E2E evidence for the affected chat cases is linked in a comment. ## Risks - Medium. The check runs once for each new local connection without credentials. On macOS it starts `netstat` and `ps` (about 30 to 80 ms). On Linux it reads `/proc`. Bearer requests do not run it. - If a process can not be identified on a supported platform, the request gets no implicit board access. On Linux, when a same-user process hides its sockets, the check keeps the previous behavior so that the operator's browser continues to work. - This is a boundary for the documented path, not a sandbox. Agent runs use the operator's OS account. A process that intentionally leaves its run's process tree, or reads the instance files directly, is outside this check. The broader option is a product decision: bind implicit board access to a capability that only the operator's browser and CLI hold, or run agents under an OS sandbox that blocks the server port. This pull request does not change `local_trusted` semantics for the operator. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
2168f50741 |
Forward provider credentials from verified image catalogs (#21186)
## Thinking Path > - Paperclip manages AI agents for work. > - Application images can ship plugins that provide task environments. > - Paperclip starts plugin workers with a restricted process environment. > - An image author can know which provider credentials an image-bundled worker needs. > - The image catalog already identifies selected bundles and verifies their content. > - This change adds a way for the image catalog to declare the provider keys that a verified worker receives. ## Linked Issues or Issue Description **What existing behavior does this improve?** Image-bundled environment drivers can use the worker credential delivery already available to built-in providers. **Current behavior** The host provides fixed credential forwarding for certain built-in providers. **Proposed behavior** A verified image bundle can receive provider API keys declared on its own catalog entry. **Reason and benefit** Image authors can scope deployment credentials to the verified worker that needs them. ## What Changed - Add an optional provider credential declaration to the image-owned distribution catalog. - Scope forwarding to the verified worker for the selected bundle. - Publish distribution catalog support for image builders. ## Verification - The new loader regression failed before the change because the old catalog rejected the credential declaration. - Ran six focused Vitest suites for bundled activation, package identity, distribution catalog validation, worker environment, and managed configuration. 166 tests passed and one existing test was skipped. - Ran `pnpm --filter @paperclipai/server typecheck`. It passed. - Reviewed the diff for secret values and personal information. Test keys are synthetic. - A downstream image build fixture exercised the catalog reader, bundle verification, and worker environment together. The supported build passed only the declared provider key to the verified worker. The build for an older target omitted the declaration. - Local verification used embedded PostgreSQL and prebuilt fixtures. ## Risks - Deploy images with catalog credential declarations only to Core versions that support the field. Older hosts reject unknown catalog fields. - The image author controls which verified bundle receives each provider API key. ## Model Used - Implementation: GPT-6.1 Sol (`gpt-6.1-sol`), delegated through Codex. Independent Codex review and verification. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs - [x] I have described the issue in the PR - [x] I have used public references - [x] My branch name describes the change - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented risks - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
267983356c |
feat(mcp): speak stateless MCP 2026-07-28 to provider servers, keep older servers working (#21140)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents use apps through Paperclip's MCP client. The client calls each provider's remote (Streamable HTTP) or local (stdio) MCP server. > - The client spoke only protocol 2025-06-18. It always did the `initialize` handshake and it sent client capabilities only in that handshake. > - MCP 2026-07-28 removes the handshake and sessions. Every request carries its version, capabilities and client identity in `_meta`. Servers that only speak 2026-07-28 reject the old requests, and they never see that Paperclip renders MCP Apps views. > - Public servers already speak it: DeepWiki and Context7 answer `server/discover` with 2026-07-28 today. > - This pull request makes the client negotiate the version per server. It speaks stateless MCP to 2026-07-28 servers and keeps the handshake for 2025-11-25 and older servers. > - The benefit is that new and old providers both work, and a tool call to a stateless server is one HTTP request with no handshake. ## Linked Issues or Issue Description Refs #21119 (MCP Apps host). This PR sends its MCP Apps capability to stateless servers too. **Subsystem affected** Server: the MCP client in `server/src/services/mcp-http.ts`, catalog discovery, the tool gateway (`tools/call`, context reads, stdio), and the remote MCP OAuth flow. **Problem or motivation** The client used protocol 2025-06-18 only. MCP 2026-07-28 makes MCP stateless: no `initialize`, no `Mcp-Session-Id`, per-request `_meta`, `server/discover`, `resultType` with multi round-trip requests (MRTR), cacheable lists, required `Mcp-Method`/`Mcp-Name` headers, and RFC 9207 `iss` validation. Servers that speak only 2026-07-28 cannot be used. Paperclip advertised MCP Apps support only in `initialize`, so stateless servers never saw it. **Proposed solution** Probe each server once with `server/discover`, pick the newest mutually supported version, and save it on the connection. Send stateless requests to modern servers and keep the handshake for legacy servers. Tool calls use the saved version and never probe. The saved version is checked again only when a request built from it is refused. **Alternatives considered** Inline negotiation (send each request in modern shape first and fall back on `400`) adds a failed request to every call to a legacy session server. Probing on every tool call adds a round trip before each tool runs. Saving the negotiated version keeps tool calls at one request. **Roadmap alignment** MCP protocol currency for the Apps and connections substrate. It follows the MCP Apps host work in #21119. **Additional context** Spec: https://modelcontextprotocol.io/specification/2026-07-28/changelog. Out of scope (separate work): the Tasks extension, Skills over MCP, `subscriptions/listen`, progress and cancellation, icons, enterprise-managed authorization. ## What Changed - `mcp-http.ts`: `negotiateMcpProtocolVersion` probes with `server/discover`. A `DiscoverResult` or a modern error (`-32020`/`-32021`/`-32022`) means a modern server. Anything else means a legacy server, which gets `initialize`. A `401`/`403` keeps the challenge for OAuth discovery. The era is remembered per connection and endpoint. - `mcpHttpRequestInit` builds every request. It always sends `Mcp-Method` and `Mcp-Name` (Base64 sentinel when unsafe). For modern versions it adds `_meta` (`protocolVersion`, `clientCapabilities` with `extensions["io.modelcontextprotocol/ui"]` and URL elicitation, `clientInfo`) and `Mcp-Param-*` headers from `x-mcp-header`. - Legacy `initialize` now requests 2025-11-25 and advertises the MCP Apps extension. - Catalog refresh saves `config.mcpProtocolVersion`. It drops tools with invalid `x-mcp-header` annotations. It reuses a fresh `tools/list` result (`ttlMs`, `cacheScope`). It probes again when the saved version is refused, and it negotiates again after a credential refresh. - `tools/call`: a saved modern version is called directly (no probe, no handshake). A modern rejection of a legacy-shaped call, or `-32022`, gets one retry with a negotiated version. The server rejected the first request before it ran, so the retry cannot repeat a side effect. A remembered legacy server that refuses `initialize` is probed again. - MRTR: `input_required` with URL requests becomes the existing provider link flow (`provider_interaction_required`). The next identical call sends `requestState` and `inputResponses`. Form, sampling and roots requests stop with `provider_input_required` and a plain message. A missing `resultType` means `complete`. - Context reads (`resources/*`, `prompts/*`) use the saved version, cache fresh results, and stop on `input_required`. The error code is kept, so resource not found (`-32602` or `-32002`) is visible. - stdio: probe with `server/discover` (2 s limit). Restart a server that exits on the probe. Send `_meta` to modern servers. - OAuth: refuse a callback without `iss` when the server advertises `authorization_response_iss_parameter_supported`. Check `iss` before acting on a provider error. Send `application_type: "native"` for localhost callbacks and `web` otherwise. Issuer-bound clients and the CIMD-first order already existed. - Docs: `doc/connections/MCP-PROTOCOL-VERSIONS.md`, linked from `doc/connections/README.md`. ## Verification - New suite `server/src/__tests__/mcp-protocol-versions.test.ts` (19 tests) uses a fake 2026-07-28 server and a fake legacy session server. It covers discovery, `_meta`, headers, version fallback, `-32022`, `resultType`, MRTR link and retry, form stop, cache TTL, legacy flow, and recovery when a saved legacy server moves to 2026-07-28. Embedded Postgres runs the end-to-end cases. - New OAuth tests in `generic-mcp-connection.test.ts`: `iss` required when advertised, `iss` checked on provider errors, `native` for localhost, and a stateless server listed again after a revoked token. - Red/green: with the master service code, all 21 new protocol and OAuth tests fail. With this change they pass. The token-refresh regression test fails without its fix and passes with it. - Existing suites: 12 touched files, 1088 tests pass. `tsc --noEmit` passes for `server`. Updated expectations count the one-time `server/discover` probe, and the Calendly DCR test now expects `native` for its localhost callback. - `pnpm verify` exits 2 (a doc is a global CI input), so CI supplies the broad coverage. - Live check with the changed client, without credentials: - DeepWiki (`mcp.deepwiki.com/mcp`): discover 200 → 2026-07-28, no session, `tools/list` 200 (3 tools). - Context7 (`mcp.context7.com/mcp`): discover 200 → 2026-07-28, no session, `tools/list` 200 (2 tools). - GitMCP (`gitmcp.io/docs`): discover 400 → `initialize` (2025-03-26) → session → `tools/list` 200 (5 tools). - Circleback (`circleback.ai/api/mcp`): discover 401 with its `WWW-Authenticate` challenge kept for sign-in. ## Risks - A legacy server that is not saved as session-required gets one extra request (`server/discover`) the first time it is refreshed. Then the version is saved. Saved session-required connections are not probed. - DCR from a localhost callback now registers a `native` client. This is what the spec asks for. Existing registrations are reused. - An authorization server that advertises `iss` support but does not send `iss` now fails sign-in with a clear message. The spec requires this. - MRTR answers are kept in memory for 30 minutes. After a restart the server asks again, and the user sees the link again. - No schema change. `config.mcpProtocolVersion` is a new key in the existing JSON config. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code with tool use (shell, file edits, web fetch of the MCP spec), extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
31c3412bbb |
perf(db): build tx.query on first read instead of for every transaction (#21148)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Every server module reaches PostgreSQL through one client,
`createDb` in `packages/db`. The server has about 1,260 `transaction(`
call sites. Run events, runner checkpoints and the run queue use a
transaction for each write.
> - Drizzle builds one relational query builder for every schema table
when it creates a database object. It creates a database object for each
transaction and for each savepoint.
> - The Paperclip schema has 321 tables. One `BEGIN` therefore allocates
about 100 KB and uses about 80 microseconds of CPU before the first
statement. One `SAVEPOINT` costs the same again. Five places read the
relational API inside a transaction.
> - On a hosted instance the server main thread was near one full core,
and garbage collection used a quarter of it. A fixed 100 KB for each
transaction is garbage that no caller asked for, and it grows with every
new table.
> - This pull request makes `createDb` build `tx.query` the first time a
transaction reads it.
> - The benefit is 82% less allocation and 7 times fewer garbage
collections for an empty transaction, and 45% less allocation for one
native run event. Transaction behavior does not change.
## Linked Issues or Issue Description
Related: #21131 (one transaction of two statements for a native run
event). That pull request reported this cost as a follow-up. No issue
exists for it.
**What existing behavior does this improve?**
The cost of `db.transaction()` and of a nested `tx.transaction()`
(savepoint) on the client that `createDb` returns.
**Subsystem affected**
Database client (`packages/db`).
**Current behavior**
`createDb` calls `drizzle(sql, { schema })`. Drizzle's `PgDatabase`
constructor then runs this loop for the root client, for every
transaction and for every savepoint:
```js
for (const [tableName, columns] of Object.entries(this._.schema)) {
this.query[tableName] = new RelationalQueryBuilder(...);
}
```
A transaction is a `PgTransaction`, which extends `PgDatabase`
(`drizzle-orm/pg-core/session.js`). `PostgresJsSession.transaction` and
`PostgresJsTransaction.transaction` construct one for each `BEGIN` and
each `SAVEPOINT` (`drizzle-orm/postgres-js/session.js`).
Measured on `master` at
|
||
|
|
82c6810107 |
fix(runtime): post the agent's blocker reply when a retried finalization blocks the task (#21169)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The native runtime finalizes an agent's run, applies the agent's status decision to the task, and posts the agent's reply as a task comment > - When the first workspace copy-back fails, or the run reconnects, a background retry commits the result instead of the live heartbeat > - That retry path applies the "blocked" status but never posts the agent's reply, so the task turns blocked with no visible reason > - The retry path already shares one recovery step for replies, but that step accepted only "done", "needs review" and conversation replies > - This pull request lets the shared recovery step present a blocked task's reply, so the retry path and the normal path converge on one presentation step > - The benefit is that a blocked task always tells the user why, exactly once ## Linked Issues or Issue Description **What happened?** An agent reports that it is blocked. The task moves to "blocked". No comment from the agent explains why. This happens when the run finishes through a retried workspace copy-back or a reconnect loop instead of the live heartbeat. The run succeeds and its result has the agent's blocker summary, but the run has no presentation decision and the task has no reply. On a staging company, 58 of the 150 most recent agent-blocked tasks for one agent had no agent comment. **Expected behavior** Every finalization path that applies an agent's status decision also presents the agent's reply. A blocked task always shows the agent's explanation once. The status decision does not post a second copy. **Steps to reproduce** 1. Run a native agent on a task. The agent reports `blocked` with a summary and a task-wide blocker. 2. Make the first workspace copy-back fail (`native_workspace_sync_out_failed`), so the heartbeat leaves finalization to the retry. 3. Let the retry finish the copy-back and commit the result. 4. The task is blocked and the run succeeded, but the task has no agent comment. **Paperclip version or commit** `master` at |
||
|
|
f1d25aca14 |
test(restart-chaos): wait for full agent termination in hosted cleanup; count only real outages in continuity (#21176)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart-chaos harness has a hosted mode (#21153) that runs the release-swap proof on a real staging instance > - Its first 2-task smoke from master graded 6 / 6 tasks completed hands-off, but cleanup ended with an error and the API continuity count showed one failure that was not an outage > - Cleanup deleted the agent as soon as its `status` read `terminated`, while its `lifecycleState` was still `terminating`; the server refused with 409 "Complete termination before deleting the agent" > - The continuity count treated a deliberate 404 (an older build without the assigned-work invariant endpoint) as a failed request > - This pull request waits for full termination and counts only real outages > - The benefit is a clean exit code after a clean run and a continuity number that means "the instance did not answer" ## Linked Issues or Issue Description Refs #21153 **Describe the bug** Hosted cleanup reports `remove agent ...: 409 Complete termination before deleting the agent` and exits non-zero after a clean run. The API continuity line counts a deliberate 404 as a failed request. **Steps to reproduce** Run the hosted smoke (`--hosted-releases 0 --hosted-tasks 2`) against an instance that does not have `/api/companies/:id/assigned-work-invariant`. **Expected behavior** Cleanup waits until the agent is fully terminated. Continuity counts only no answer, server errors, and 401, 403, 408 or 429. ## What Changed - `hosted-setup.ts`: agent removal waits until both `status` and `lifecycleState` are `terminated` before the delete. - `hosted-api.ts`: a 400, 404, 409 or 422 answer is a deliberate refusal, not a continuity failure. No answer, 5xx, 401, 403, 408 and 429 still count. - `hosted-main.ts`: `finalizeHosted` passes the poll interval through, for the test. - `README.md`: states the continuity rule. ## Verification - `node server/node_modules/vitest/vitest.mjs run --config tests/restart-chaos/vitest.config.ts tests/restart-chaos/hosted-*.test.ts`: 34 tests pass. The setup-failure cleanup test now returns `terminated` with `lifecycleState: terminating` first and checks that the harness reads again before it deletes. The continuity test checks that a 404 is not a failure and a 401 is. - `node server/node_modules/typescript/bin/tsc -p tests/restart-chaos/tsconfig.json`: clean. - Live: the smoke `rc-hosted-smoke2` on a hosted staging instance hit the 409. A second `--hosted-cleanup-only` found nothing left: the agent is terminated and the product keeps its record (cost history), its tasks are hidden, its secret is deleted and its project is archived. ## Risks - Low risk: harness-only. Cleanup can wait up to 120 s per agent for termination, as before. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits). Default context window; no extended thinking mode. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
5412162716 |
test(restart-chaos): grade staging-mix hands-off and click Retry once after the settle (#21168)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness grades agent tasks across server swaps, and the staging-mix workload (#21125) is its staging-shaped load > - On a fix-PR comparison the harness clicked Retry as soon as the banner appeared (once after 11 s), which pre-empted the product's own continuation > - So the wave could not show whether a task would have continued by itself, and the banner itself went uncounted > - This pull request makes staging-mix grading hands-off by default: no Retry click during the wave or the settle, one click only after the settle, and every banner recorded > - The benefit is three honest numbers per wave: completed hands-off / started, Retry banners shown, and Retry worked / clicked ## Linked Issues or Issue Description Refs #21125, #21126, #21139. **What existing behavior does this improve?** Staging-mix Retry handling and the `done_comment_once` check in the restart chaos harness. **Subsystem affected** Test harness only: `tests/restart-chaos/`. No product code changes. **Current behavior** The observer clicks Retry as soon as a task rests with the banner, so a product continuation that would have come is never seen. A banner that heals by itself is not recorded. `done_comment_once` counts every agent comment that mentions `DONE-<nonce>`. A closing summary that quotes the marker therefore failed all 6 tasks of wave `w-20261112T174255-s7101`. The release-swap grader can click a second Retry after the observer's. **Proposed behavior** Hands-off by default. Every banner is recorded with its first-shown time, its duration and its error code. One Retry click happens only after the settle, and only for a task that is still not done and that nobody clicked. The DONE check counts only the comment whose whole body is the marker. A completion summary repeated by a different run becomes its own `completion_once` finding. **Reason and benefit** The product goal is that the Retry banner never appears and that interrupted work continues by itself. The wave now measures exactly that. ## What Changed - `staging-mix.ts`: - The observer step records the Retry banner per task (`retryBanner`: first shown, total shown time, error code). It clicks during an overlap only with the new flag `--mix-retry-during-wave`. - `isDoneComment` counts only the bare `DONE-<nonce>` comment. - `completion_once` fails when more than one run posted a comment that mentions the marker. - Comments are now collected with their run ids. - `observer.ts`: - A staging-mix task with a banner is kept under watch instead of being clicked or rested (hands-off). - `retryAfterSettle` reopens a task for the one post-settle click. - New `staging-mix-retry.ts`: - `shouldClickRetryAfterSettle` is the single click rule. It never clicks a done task, a task with a scheduled retry, or a task already clicked. - `retryAfterSettleStage` waits out the settle (new flag `--mix-settle-minutes`, default 15 after the last fault slot ends), clicks once per stuck task, and follows each click to rest. - The results module writes `staging-mix-results.json` and `staging-mix-summary.md`. Their first three lines are `completed hands-off / started`, `Retry banner shown: N tasks`, `Retry worked / Retry clicked`. Then come the banner list and one end-of-wave read of `GET /api/companies/:id/assigned-work-invariant` (`violationCount`, `violations`, `agentsWithStalledSlots`, `openTasks`). - `release-swap-finish.ts` uses the same click rule, so the swap grader never clicks a task the observer already clicked. When the release-swap cell drives the wave, the staging-mix settle stage does not run; the swap grader makes the one click. The hosted mode can import the same function. - `main.ts`: the staging-mix end-of-wave report moved into `StagingMixRun.report`. The file shrinks from 1,000 to 998 lines. - README: the Retry and results sections, the two new checks, and the local short-form command (without `--ephemeral-server`, which runs only on Daytona). ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 59 files, 714 tests pass. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: passes. - New tests cover: - the default and opt-in flags; - the click rule; - a settle where one task heals by itself (no click), one task gets exactly one click, and one already-clicked task gets none; - the three headline numbers, including the swap grader's clicks; - the invariant read; - the DONE count against the real comments from `w-20261112T174255-s7101` t001 (summary plus bare marker passes; a summary repeated by another run fails `completion_once`). - Not run: a live wave with this change. The Daytona image for current master is still queued. ## Risks - Low risk. Test harness only. - A hands-off task that keeps its banner now waits to its deadline before the post-settle click, so a wave with many banners runs longer. - RS1 waves on other workloads no longer get a second end-of-wave click on a task the observer already clicked. That is intended. - Known master CI failures, not touched here: the storage S3 multipart throttle test (being fixed) and the Runner native-distribution-integrity test. - `ci / Runner support suite` failed on head |
||
|
|
7f69ef8b44 |
Prepare and finalize managed workspaces on the active Launcher (#20743)
## Thinking Path > - Paperclip lets plugins own managed Git workspaces. > - A portable Runner task can move to a different physical allocation. > - The checkout must be present on the allocation that launches the Runner. > - Workspace preparation and finalization need exact placement when they retry. > - This change checks the placement at Runner admission and records it before finalization. ## Linked Issues or Issue Description **What existing behavior does this improve?** Workspace preparation and finalization for portable Runner tasks. **Current behavior** A task replacement can change the allocation after Git preparation. A lost finalization reply can leave retry without the original placement. **Proposed behavior** Prepare the checkout on the actual allocation before admission. Record finalization placement so an exact retry can use it. Portable ownership and controller recovery are in [#21066](https://github.com/paperclipai/paperclip/pull/21066). ## What Changed - Reserve the managed workspace before acquiring the portable task lease. - Pass the task placement to the workspace provider. Reuse the described task while placement stays the same and prepare again before admitting a replacement. - Record the initial preparation's run, agent, generation, and optional task placement before dispatch. Inspection and retries on the same Launcher replay the same request; VM and sandbox preparation use the same recovery path. - Keep provider request identity separate from Git credential authority: reservation reconciliation uses the existing project-consent path in a host-only scope bound to the plugin and workspace. Foreground preparation still requires an active run; consent, repository, and revocation checks are unchanged. - Acquire the current Launcher before retrying preparation. If placement changes, use ordinary preparation on the replacement so the provider can settle the prior writer and recover the retained checkout. No additional provider discovery call is added. - Select finalization placement before publishing the operation record, then dispatch and recover using that saved choice. Retain the Launcher during bounded finalization retries. - Preserve interrupted pre-upgrade finalization requests without placement, including their original operation IDs and provider routing. No migration or new durable mode flag is required. ## Verification - Final revision `74ea3e6712`: server typecheck and exact SDK source-pin verification (538 inputs) passed. - All 185 tests passed across the full plugin worker manager, project Git authorization, and plugin workspace integration suites (including 56 workspace tests). Independent verification reran the three lost-reply cases: inspection, same-Launcher retry, and replacement after deleting the original Launcher. The tests assert the correct request identity, durable intent, and clearing that intent after success. - Before the final retry-accounting fix, all 323 tests passed across workspace admission, runtime execution, native workspace finalization, plugin workspace integration, and terminal cleanup retry. Regressions exercise crashes after operation insertion but before dispatch in both finalization paths, exact replay after lease release, and retry after ambiguous placement clears without publishing an incomplete operation. Pre-upgrade interrupted requests retain their operation IDs despite an available portable lease. - Final retry-accounting fix: all 84 native workspace finalizer, failure, and supersession tests passed. A real missing-plugin regression first failed on the prior commit, then verified healthy-run progress, three-attempt exhaustion, terminal cleanup, and absence of a misleading workspace operation. Independent verification also passed the legacy and crash-before-dispatch cases. - Worker-process tests cover both preparation RPC methods: the provider still receives the original run and agent, maintenance receives project-scoped credential authority, and foreground or mismatched-scope requests remain denied. Database tests confirm a stopped run cannot authorize foreground Git access while valid project consent can authorize maintenance. - Existing metadata protection coverage proves callers cannot overwrite the private preparation intent or read it through workspace responses. - Two existing Launcher Git tests passed: retry preserves dirty files and the original base commit without fetching, and retry resumes the saved checkout intent while rejecting a changed pending branch. - Implementation was delegated to GPT-6.1 Sol and independently reviewed. The GitHub stack's automatic rebase preserved the original workspace-placement patch. - [Full CI](https://github.com/paperclipai/paperclip/actions/runs/43293046205/job/131452117706) passed on `74ea3e6712`, including server and workspace tests, Runner, end-to-end tests, typecheck, build, and canary dry run. Ordinary Greptile and [Apex](https://github.com/paperclipai/paperclip/pull/20743#issuecomment-6788920388) both scored `74ea3e6712` 5/5, with no unresolved findings. ## Risks - Live Daytona end-to-end qualification is pending. The checkout record persists on the mounted volume; replacement preparation relies on the provider confirming the previous writer is settled before using that retained checkout. Local tests cover Core request routing and Launcher's durable checkout recovery separately. ## Model Used - OpenAI Codex GPT-6 with GPT-6.1 Sol agents for implementation and independent review. ## Checklist - [x] Thinking path and model are included. - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] The issue is described using the enhancement template. - [x] Local checks passed and new behavior has tests. - [x] All Paperclip CI gates are green. - [x] Greptile is 5/5 with no open findings. --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
ea400ecb07 |
chore(lockfile): refresh pnpm lockfiles (#21178)
## Thinking Path > - Paperclip uses a dependency lock for reproducible builds. > - Dependency changes on master need matching lock entries. > - Paid eval reports also pin a reviewed offline correction. > - Reports must install before paid model work starts. > - This automated PR keeps the lock and report proof together. ## Linked Issues or Issue Description **What happened?** Dependency manifests on master changed after the last committed lock refresh. **Expected behavior** Frozen installs must match those manifests. Eval reports must retain matching offline lock proof. **Steps to reproduce** Run a frozen install before and after this lock refresh. ## What Changed - Refresh the dependency lock from the manifests on master. - Refresh the sandbox provider lockfiles from their manifests on master. - Include the matching offline report correction and hashes. - Resync the SDK public source proof's lock pins from the refreshed lock. ## Verification - Pinned pnpm resolution completed with lifecycle scripts disabled. - The report proof refresh accepted the reviewed committed baseline. - The workflow rejected changes outside the lock and report proof files. ## Risks The lockfile bot merges this PR without CI or review when every added package version is at least 3 days old on npm. Otherwise the PR waits for a maintainer, and the daily refresh tries again. The bot merges only the exact commit this workflow generated from manifests already reviewed on master. Unknown report lock baselines remain blocked. This update changes no permissions or paid enable gates. ## Model Used None — generated by pnpm and the deterministic report-lock updater. ## Checklist - [x] Thinking path and problem description included - [x] Generated lock and matching report proof included - [x] Release-age gate decides between bot merge and maintainer review Co-authored-by: lockfile-bot <lockfile-bot@users.noreply.github.com> |
||
|
|
8bff629465 |
fix(environments): skip blank lease ID queries (#15200)
## Thinking Path > - Paperclip manages work for AI agents. > - Runs can use environment leases. > - A lease lookup sends its ID to a PostgreSQL UUID column. > - An empty ID makes that query fail on every poll. > - This pull request returns null before querying for blank IDs. > - Valid lease IDs keep the existing lookup behavior. ## Linked Issues or Issue Description Fixes: #15143 ## What Changed - Skip database access for empty or whitespace-only lease IDs. - Cover empty, space, and newline IDs with a database-query spy. - Keep an unknown valid UUID returning null. Existing lease tests cover valid lookups. ## Verification - The branch was rebased onto current master `252d9b50` to resolve the test-file conflict. - All 53 environment service tests passed on Node 24.19.0. The run used cached dependencies and current database and shared-package source aliases. - The blank-ID cases assert that no database query occurs. The unknown valid UUID still returns null. The existing suite covers stored leases. - `git diff --check`: passed. - A clean frozen install is blocked because upstream master adds two OpenTelemetry packages without updating the lock file. - Full typecheck and build were attempted on both unmodified master `252d9b50` and this head in the same environment. Both fail with the exact same runner `SessionRecord.cursor_prompt_usage` error (TS2339 at `codex-runtime-adapter.ts:1572`). Full validation is pending CI on the rebased commit. - The earlier full test run had a Slack callback ordering failure that also occurred on unmodified master. - Greptile reviewed the previous head at 5/5 with no inline findings. ## Risks Low risk. Only blank IDs return early. Non-blank IDs still use the database. There is no schema or migration change. ## Model Used OpenAI Codex, based on GPT-6, assisted with source review, implementation and local test execution. The exact deployment model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (targeted suite; full suite pending above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes (no user-facing configuration or command changes) - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green (new head checks are pending) - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge |
||
|
|
a37f3056b6 |
perf(docker): keep dependencies and provider binaries in layers that survive a commit (#21167)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Each merge to master builds the production Docker image for amd64 and arm64 and pushes it to GHCR > - Each build pushes three layers again, about 3.5 GB compressed of a 4.7 GB image > - A file-by-file comparison of two consecutive master images shows that about 95% of those bytes do not change between commits > - The unchanged bytes are in the same layers as small files that change on every commit, so the registry sees new layers > - This pull request moves the unchanged bytes into their own layers > - The benefit is a smaller push for each master build and a smaller pull for each deploy ## Linked Issues or Issue Description **What existing behavior does this improve?** The production image build. It pushes about 3.5 GB of layers for each architecture on each master commit. Each deploy pulls the same bytes again. **Subsystem affected** Docker image build (`Dockerfile`, `production` and `runner-provider-pack` stages). **Current behavior** I compared the amd64 images of two consecutive master commits (`2174f2d` and `dfc510b`, 15 changed server source files). I hashed each file in the three layers that changed. | Layer | Compressed | Content that is identical between the two builds | | --- | --- | --- | | `COPY --from=build /app /app` | 1,404 MB | 97.5%. The root `node_modules` is 2.86 GB of 4.13 GB and has 0 changed files. | | `COPY /provider-pack` | 1,008 MB | 100% of the 112 files of 1 MiB or more (2.54 GB). Only 32 small files change (manifest, `.modules.yaml`, `.bin` shims). | | `COPY /provider-pack-transport/` | 1,059 MB | The archives differ only in tar timestamps. This pull request does not change this layer. | **Proposed behavior** - Copy the root `node_modules` from the `deps` stage as its own layer. Copy the remainder of `/app` from the `build` stage with `--exclude=node_modules`. - In the `runner-provider-pack` stage, move pack files of 1 MiB or more to `/provider-pack-large` after the transport archive is made. Set their mtimes to a fixed value. - Copy `/provider-pack-large` with `COPY --link`, then copy the small remainder of the pack to the same directory. **Reason and benefit** The dependency layer changes only when the lockfile changes. The large provider-pack layer changes only when a provider version changes. The expected push for a commit that changes source only goes from about 3.5 GB to about 1.5 GB for each architecture. The master build runs about 140 times a day on AWS runners, and this traffic is about 570 GB a day of billed data transfer. Deploy pulls become smaller by the same amount. **Breaking changes** None. The file tree of the image is the same. Provider-pack files of 1 MiB or more now have an mtime of 1970-01-01. ## What Changed - `Dockerfile`: the `production` stage copies `/app/node_modules` from `deps` first, then `/app` from `build` without the root `node_modules`. - `Dockerfile`: the `runner-provider-pack` stage splits large pack files into `/provider-pack-large` with fixed mtimes. The `production` stage copies that tree with `--link` before the remainder of the pack. - `server/src/__tests__/docker-image-layer-reuse.test.ts`: a drift guard for the copy order, the exclude, the `--link`, and the position of the split after the archive step. ## Verification - `cd server && npx vitest run src/__tests__/docker-image-layer-reuse.test.ts src/__tests__/docker-build-stamp.test.ts`: 9 tests pass. The 2 new tests fail against the `Dockerfile` on master. - `node scripts/check-docker-deps-stage.mjs`: PASS. - `pnpm verify`: exit 2 (the `Dockerfile` is a global CI input). Required CI supplies the broad coverage. - A small Dockerfile with the same syntax version (`docker/dockerfile:1.20`) and the same four `COPY` instructions, built two times with a changed stamp file: - `--exclude=node_modules` leaves out only the root `node_modules`. A nested `node_modules` stays. - The dependency layer has the same digest in the two builds. - The large-file layer has the same digest in the two builds only with `--link` and fixed mtimes. Without one of them the digest changes. - Hard links between large files stay hard links. The merged directory contains each file one time. - Evidence that the `build` stage writes nothing to the root `node_modules`: in the published image, all 107,413 files there have an mtime in the minute of the `deps` install. The app build ran 15 hours later. **Not verified:** a full build of the production image with this `Dockerfile`. The local Docker disk was full, and no pre-merge workflow builds this image. The first full build is the master build after merge. The check after merge is to compare the layer digests of two consecutive master images. ## Risks - Medium until the first master build. If the image build fails, master image builds stop until a revert. No broken image can be published, because the build fails before the push. - If a future build step writes to the root `node_modules` in the `build` stage, the image will not contain that change. The comment in the `Dockerfile` states this rule. - A file that crosses the 1 MiB boundary moves between the two pack layers. This only changes which layer is pushed. - The first build after merge pushes all the new layers one time. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, in Claude Code with tool use (shell, AWS CLI, Docker, registry API). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
eaef46b60b |
feat(ui): browser page-load tracing to Honeycomb for hosted instances (#21156)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The board UI is the main surface. Its load speed is a product requirement. > - The team wants to track page-load times over time and give each metric an owner and an SLO in Honeycomb. > - Server traces show only the server part of a page load. They do not show JavaScript start-up, lazy route code, rendering, or client-side navigation. > - Paperclip is also self-hosted by others. Browser tracing must stay off there, and it must not slow any page. > - This pull request adds real-user browser tracing for hosted Paperclip Cloud instances only. It loads lazily and sends a closed set of attributes. > - The benefit is one `page.load` measurement per page (named by route pattern), plus Web Vitals and API spans that join the server trace, ready for Honeycomb queries and SLOs. ## Linked Issues or Issue Description No public issue exists. Feature description: **Subsystem affected** Board UI (`ui/`), the authenticated session route (`server/src/routes/auth.ts`), and the shared session schema. **Problem or motivation** We cannot measure how long the all-tasks page or a task page takes to load for real users. Server traces stop at the API response. Most navigation is client side, and the standard OpenTelemetry document-load instrumentation does not cover it. Without a per-page browser measurement, no agent can own a page-load SLO. **Proposed solution** The server sends a `browserTracing` config in the authenticated session only when the instance is cloud managed and `PAPERCLIP_BROWSER_TRACES_ENDPOINT` is set. The browser then measures every hard load and client navigation in one shared recorder tied to the router's own route tree. After the page settles, it lazily loads an OpenTelemetry chunk that exports `page.load`, `web_vital.*`, and same-origin `/api/` fetch spans to Honeycomb. An allowlist removes every other attribute before export. **Alternatives considered** `@honeycombio/opentelemetry-web` was the first choice. Its defaults add the full page URL, query string, hash, referrer, and user agent to every span and to the resource, and it bundles log and metric pipelines. We would have to replace its exporter and remove most of its defaults. Plain OpenTelemetry web packages plus `web-vitals` are smaller and send only what we choose. Honeycomb needs only OTLP/HTTP and its ingest-key header. **Roadmap alignment** This is internal hosted-instance observability. It changes no product behavior and no self-hosted behavior. ## What Changed - `server/src/browser-tracing-config.ts`: resolves the browser tracing config. It returns `null` unless `isCloudManagedInstance()` is true and `PAPERCLIP_BROWSER_TRACES_ENDPOINT` is set. It accepts only a Honeycomb ingest key (`hcaik_` prefix), requires `https` (except loopback), and fails closed with one value-free warning on any invalid setting. Optional: `PAPERCLIP_BROWSER_TRACES_INGEST_KEY`, `_SAMPLE_RATE` (default 1), `_SERVICE_NAME` (default `paperclip-web`), `_ENVIRONMENT` (default `SENTRY_ENVIRONMENT`). - `GET /api/auth/get-session` adds `browserTracing` (config or `null`), the same delivery path as the front-end Sentry DSN. The shared `authSessionSchema` has the new optional field `browserTracingConfigSchema`. - `ui/src/lib/browser-tracing/page-load-recorder.ts` and `controller.ts` (main bundle, no OpenTelemetry imports): one `page.load` measurement per hard load (from navigation start) and per client navigation (from the history change). A measurement ends at the first painted frame where no full-page loading state is mounted and either every page-declared content signal is ready or no query without data is fetching. Redirects continue the measurement. A new navigation interrupts it. 30 s times out. The sample decision is made once per page session before any download. - `ui/src/components/PageLoadTracker.tsx`: `App` passes the same `<Route>` tree it renders, so every page is reported by route pattern (for example `/:companyPrefix/issues/:issueId`) with no per-page code. - `usePageContentReady` hook and blocking gates: `PaperclipLoading` (also the route `Suspense` fallback) blocks readiness. The task page uses the existing `issue-detail:navigate→content-paint` point. The plain all-tasks list signals when its first rows render. - `ui/src/lib/browser-tracing/sdk.ts` (lazy chunk, loaded after the page settles): OpenTelemetry web tracer, OTLP/HTTP exporter, fetch instrumentation for this origin's `/api/` only with `traceparent` sent only to this origin, `web-vitals` LCP/INP/CLS, and a context manager that makes the open `page.load` span the parent of its API calls. `span-sanitizer.ts` applies the closed attribute allowlist (no URLs, ids, query strings, status messages, or event attributes). Uncaught errors are not sent because Sentry already reports them. - New `ui` dev dependencies, pinned exactly. Each is needed, and the OpenTelemetry ones are the same versions the server already resolves: - `@opentelemetry/api@1.9.1` (already in the workspace): tracer and context API. - `@opentelemetry/sdk-trace-web@2.10.0`: the browser tracer provider, batch processor, and stack context manager. - `@opentelemetry/instrumentation-fetch@0.221.0`: fetch spans and `traceparent` injection. - `@opentelemetry/resources@2.10.0` (already a server dependency): the resource. - `@opentelemetry/otlp-transformer@0.221.0` (already a server dependency): OTLP JSON serialization for a 60-line `OtlpFetchExporter`. This replaces `@opentelemetry/exporter-trace-otlp-http`, which is a server optional peer: installing it in the workspace made the server's exact-version gate test see it as present. - `web-vitals@6.2.3`: LCP, INP, and CLS. All of them are in the lazy chunk only. The lockfile is not committed: `pr-trusted` rejects lockfile edits in pull requests, CI installs with `--no-frozen-lockfile` after the frozen attempt reports `ERR_PNPM_OUTDATED_LOCKFILE`, and the lockfile bot refreshes it after merge. - `tests/runner-e2e/source-sdk-public-inputs.json`: refreshed pins for the three shared files (the session schema is part of the pinned public SDK surface; the contract must live in `packages/shared`). - Docs: new `doc/browser-performance-tracing.md` (enablement, config, key handling, span and attribute list, page-load definitions, example Honeycomb queries and an SLO SLI). Pointers in `doc/observability.md` and the Honeycomb tracing skill. - Two App routing tests mock `PageLoadTracker` because they render `App` without a `QueryClientProvider`. ## Verification - `pnpm verify` returns exit 2 (broader verification required) because docs are global CI inputs. Required CI supplies the broad coverage. - Focused local tests, all pass: - `ui`: `vitest run src/lib/browser-tracing src/components/PageLoadTracker.test.tsx src/components/SentryGate.test.tsx src/api/auth.test.ts src/lib/issue-detail-performance src/pages/IssueDetail src/pages/Issues src/App src/components/CloudAccessGate.test.tsx src/components/IssuesList`: 17 files, 484 tests. - `server`: `browser-tracing-config.test.ts`, `auth-routes.test.ts`, `sentry.test.ts`: 63 passed, 5 skipped. - `packages/shared`: `validators/access.test.ts`: 19 passed. - New tests prove: the SDK module is never imported when the session config is `null`, absent, or signed out, or when the session is not sampled; route templates contain no ids, slugs, or tokens; a client navigation produces a `client` measurement under `/:companyPrefix/issues/:issueId`; `traceparent` goes only to this origin's `/api/` and not to a third-party or non-API URL; exported spans carry only allowlisted attributes. A mutation check (tracker forced on) turned the "never imports" tests red. - Typecheck: `ui` (`pnpm typecheck`), `server` (`pnpm --filter @paperclipai/server typecheck`), and `packages/shared` pass. - Real-browser run: a local dev server with a local OTLP collector and headless Chromium. The self-hosted session returned `browserTracing: null` with the endpoint set, and the page made 0 SDK and 0 collector requests. With a cloud session config injected into the session response, the collector received a hard `page.load` (route `/:companyPrefix/issues`, `ready_signal=idle`), a client `page.load` to `/:companyPrefix/issues/:issueId` (`ready_signal=content`), a back navigation, LCP/INP/CLS spans, and `/api/` fetch spans parented to the open `page.load` span. No task identifier, company id, task title, host, or URL appeared in the exported payload. No third-party request carried `traceparent`. - Bundle size (`ui` production build, initial JavaScript closure of `index.html`): before 3,813,986 B raw / 1,153,991 B gzip; after 3,820,743 B / 1,156,590 B (+6.8 KB raw, +2.6 KB gzip, +0.2%). The lazy SDK chunk is 84 KB raw / 27 KB gzip and is not fetched unless tracing is on and sampled. - Review fixes (each red on the old code, green on the new): the SDK now loads by a 35 s deadline even if the page never settles, so a timed-out hard load is exported (`controller.test.ts`); an agent chat with no saved task uses the idle rule instead of a content gate that never fires (`usePageContentReady.test.tsx`); the sample rate is rounded to exactly 1/N so Honeycomb's `SampleRate` weight matches (`browser-tracing-config.test.ts`). The real-browser run was repeated after the exporter change with the same result. ## Risks - Low risk for self-hosted installs: the server never sends a config, so no tracing code beyond the small recorder runs, and the SDK chunk is never fetched. - On hosted instances the feature is off until an operator sets `PAPERCLIP_BROWSER_TRACES_ENDPOINT` and an ingest key. When on, the controller wraps `history.pushState`/`replaceState` to time navigation starts and restores them on sign-out. - Every signed-in browser can read the ingest key, like the Sentry DSN. Only Honeycomb ingest keys (send-only) are accepted. A collector endpoint can keep the key server-side. - Hard-load API calls happen before the SDK loads, so they have no fetch spans. The hard-load `page.load` span is still recorded. - The browser-to-server trace join needs server tracing. The hosted image does not install the OpenTelemetry server peers today (`CLOUD_BUNDLED_SERVER_DEPS` is `@sentry/node` only). Browser spans work without it. - `ui/src/pages/IssueDetail.tsx` was already over 1,000 lines. This change adds 5 lines there for the task page content signal, next to the existing content-paint measure. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) via Claude Code, with tool use (shell, file edits, local browser automation). Reasoning effort: low. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
8990e94005 |
fix(ci): keep community PR runs from delaying CODEOWNERS jobs in the hosted queue (#21166)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Its pull request CI runs on two pools: an AWS fleet for CODEOWNERS, and GitHub-hosted runners for every other contributor > - Some short jobs for CODEOWNERS pull requests also run on GitHub-hosted runners: the commitperclip review and the merge review notifier > - GitHub starts about 550 hosted jobs each hour for this repository, and all hosted jobs wait in one first-come queue > - A community pull request run puts 52 jobs in that queue at one time, so the short CODEOWNERS jobs wait behind them > - This pull request limits how many jobs one hosted run can queue, and lets the short CODEOWNERS jobs use the reserved AWS metadata fleet > - The benefit is that community pull requests do not delay CODEOWNERS work ## Linked Issues or Issue Description **What happened?** On 2026-11-12 the repository had about 348 queued workflow runs. Jobs on the AWS fleet started in a median of 29 seconds. Jobs on `ubuntu-latest` did not: | Job | Median wait | Job length | | --- | --- | --- | | PR merge review notification | 42 min | 9 s | | commitperclip PR Review | 36 min | 15 s | | Community PR test jobs | 57 min (p90 76 min) | 7 min | Between 12:00Z and 18:00Z, hosted job starts stayed between 492 and 567 each hour while the number of running hosted jobs changed from 63 to 134. The start rate is the limit. Community PR runs supplied 3,462 of the 3,744 hosted jobs in that window (92%). **Expected behavior** A short job for a CODEOWNERS pull request starts in about one minute, whatever the community pull request load is. **Steps to reproduce** 1. Open several pull requests from forks in a short time. 2. Open a pull request as a CODEOWNER. 3. Look at the start time of `commitperclip PR Review` and `PR merge review notification` for the CODEOWNER pull request. ## What Changed - `pr-trusted.yml`: each of the four test matrices has `max-parallel: 4` when the run uses `ubuntu-latest`. The run has the same jobs. The other jobs wait in the run, not in the shared queue. AWS runs have no limit (256). - `commitperclip-review.yml` and `pr-merge-review.yml`: the job uses the reserved `paperclip-ci-routing-x64` fleet when `AWS_CI_SHORT_JOBS_ENABLED` is `true`, the pull request branch is in this repository, and the author and the sender (or the run actor and the triggering actor) are in the AWS admission list. The admission variables are the same as in `ci-routing.yml`. All other events use `ubuntu-latest`. - Check-run and commit-status notifications stay on `ubuntu-latest`. Those events do not identify the author. - `ci-routing.test.mjs`: 24 placement cases for the two jobs, and a contract test for the matrix limit. - `doc/DEVELOPING.md`: the two settings and the runner group step. ## Verification - `node --test .github/scripts/tests/ci-routing.test.mjs .github/scripts/tests/notify-pr-merge-review.test.mjs .github/scripts/tests/ci-plan.test.mjs .github/scripts/tests/ci-profile.test.mjs .github/scripts/tests/run-quality-gates.test.mjs`: 200 tests pass. - `actionlint` accepts the three workflows, including the `max-parallel` expression. - Not verified before merge: `pr.yml` calls `pr-trusted.yml@master`, and the two short workflows run from master. The effect on queue time can only be measured after merge. ## Risks - A community pull request run takes longer when the hosted queue is empty: about 45 minutes for the server matrix, compared with about 13 minutes. Today these runs take 150 to 200 minutes because of the queue. - The AWS routing does nothing until a maintainer sets `AWS_CI_SHORT_JOBS_ENABLED`. Before that, the `paperclip-ci-routing` runner group must allow the two workflow references in `doc/DEVELOPING.md`. If the variable is set first, the two jobs for CODEOWNERS pull requests stay queued. - The two jobs run only scripts from master and no pull request code. The secrets they read do not change. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
edf7edac60 |
test(restart-chaos): keep wake evidence and early run events, and word staging-mix rounds unambiguously (#21173)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness runs agent tasks across server swaps and keeps evidence for each failure > - Wave `w-20261112T181948-s7101` had two "run started after the task was done" cases that could not be traced > - The wave deleted its database before anyone saved which wake started those runs, which server created them, or how the task status changed > - The evidence bundle also kept only the last 200 events of a run, so it dropped the first 291 events of a Retry run, including its wake payload > - The staging-mix wording made one Codex agent refuse to redo an interrupted round, and the task blocked on a harness artefact > - This pull request saves that evidence for every task, keeps a run's first events, and rewords the round rules without changing the grade ## Linked Issues or Issue Description Refs #21125, #21168. **What existing behavior does this improve?** Restart chaos evidence capture and the staging-mix task wording. **Subsystem affected** Test harness only: `tests/restart-chaos/`. No product code changes. **Current behavior** - The harness does not save a task's wake rows, run context snapshots or status history before the wave database is deleted. - `evidence.ts` keeps only the last 200 events per run. - The task tells the agent "do not skip, repeat or shorten any" round, and the follow-up says "do not restart any". An agent read that as "never re-run an interrupted round". **Proposed behavior** - Save, for every task, the wake rows, run snapshots, status changes and serving server generations. - Keep a run's first 400 and last 200 events, with a marker between them. - Say plainly that every round must appear in the ledger exactly once, that an interrupted round must be run again, and that a round already in the ledger must never be run again. **Reason and benefit** Wake and run origins can be traced after the wave, and the wording no longer creates failures that come only from the harness. ## What Changed - New `wake-evidence.ts`, called once at the end of each wave while the database still exists. It writes `wake-evidence/<task>.json` for every task. Each file holds: - for each run: its `context_snapshot`, invocation source, trigger and wakeup request id; - the `agent_wakeup_requests` rows for the issue and for each run: reason, source, trigger detail, requesting actor, idempotency key, payload and times; - the issue's status changes from `activity_log`. Each wake and run names the server generations alive when it was created (`servingAtCreate`): pid, boot id, epoch and process space from `controller_instances`, plus the harness's primary or secondary role during an overlap. Each run also names its owning controller (`owner`). `wake-evidence/servers.json` lists every controller and harness server window. Known secret values are redacted. - `evidence.ts` keeps all of a run's events, or the first 400 and the last 200 with an `omittedEvents` marker between them. - `staging-mix.ts` only changes the wording; the grading is unchanged. The new text: - every round must appear in the ledger file exactly once, with the hash the command printed; - before a round, read the ledger; a round whose line is there is finished and must never run again; - a round interrupted before its line was written must be run again. The follow-up comment says the same. - `main.ts` gains one call; the file stays at 1,000 lines. README updated. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 59 files, 711 tests pass. That includes a real embedded PostgreSQL test that writes a wake, a run, a status change and a controller row, then checks the evidence file. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: passes. - Unit tests cover the serving-generation join during an overlap, the per-issue join, the first 400 and last 200 events of a 1,291-event run, and the new wording. - Not run: a live wave with this change. ## Risks - Low risk. Test harness only. - Evidence capture is one set of reads per wave. A failure is recorded as `wake_evidence_failed` and does not fail the wave. - Known master CI failures, not touched here: the storage S3 multipart throttle test and the Runner native-distribution-integrity test. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, file edits). Extended reasoning was available. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
4363a3115a |
fix(recovery): continue any unfinished run in a fresh run from saved work (#21135)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Agent runs are interrupted when a server restarts, a deploy swaps
servers, a controller lease is lost, or a sandbox Runner cannot be
reached again
> - Exact recovery (resume the same session, replace, retry) refuses in
many ways, and each refusal leaves the task blocked or idle for a person
> - On staging, 577 of 1,000 runs in 11 hours failed or were
interrupted, across 25 error codes. About 900 earlier PRs each fixed one
refusal code, and the rate did not converge
> - This pull request adds one universal fallback: when an unfinished
run cannot be resumed exactly, for any reason that is not the person's,
the task continues in a fresh run from its last saved workspace and its
conversation
> - The benefit is that a new failure code next week falls back without
code changes, and tasks stop waiting on a person for infrastructure
faults
## Linked Issues or Issue Description
**What happened?**
After a server restart or deploy swap, many runs end with codes such as
`native_continuation_requires_reconciliation`
(`runner_remote_process_identity_unavailable`), `controller_lease_lost`,
`process_lost`, `server_shutdown_interrupted` and
`native_session_interrupted` (including `runner_harness_state_mismatch:
resumed_not_suspended` / `resumed_unavailable`). The disposition worker
then settles the recovery hold as "blocked, no replay". The task stays
Blocked until a person presses Retry or reconciles the run.
**Expected behavior**
A task waits on a person only for reasons that are the person's: an
approval, a question, a credential to share, a budget stop, a provider
usage limit, or a Stop they asked for. For every other failure, the
system starts a new run for the same task from the last verified saved
workspace, without a person.
**Steps to reproduce**
1. Start a task on a sandbox agent.
2. Restart or swap the server while the run is in progress, so the new
server cannot re-reach the Runner.
3. See that the run fails with
`native_continuation_requires_reconciliation` and the task stays Blocked
with no successor run.
**Paperclip version or commit**
`master` at
canary/v2026.1112.0-canary.15
|
||
|
|
8ff45c0fa9 |
feat(connections): render MCP Apps views in the task chat, starting with Circleback (#21119)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Agents reach external apps through MCP connections. Paperclip
records each call and shows it as a tool row in the task chat.
> - MCP Apps (SEP-1865, extension `io.modelcontextprotocol/ui`, spec
2026-01-26) lets an MCP server attach an interactive `ui://` view to a
tool. Claude, Claude Code and VS Code render these views in chat.
Circleback uses them to show meeting cards.
> - Paperclip dropped the tool's `_meta.ui` at discovery and the
result's `_meta` at call time. Its gateway reported MCP App UI as
"unsupported". Circleback was reachable only through Composio, whose
toolkit has no `Display*` tools.
> - This pull request makes Paperclip an MCP Apps host. It also adds
Circleback as a native catalog connection.
> - The benefit is that a Circleback recap in a task shows the same
meeting cards as in Claude, and any other MCP server with views works
the same way.
## Linked Issues or Issue Description
**Subsystem affected**
Connections and apps (remote MCP gateway) and the task chat.
**Problem or motivation**
An agent that uses Circleback in Paperclip can only return text. The
same tools in Claude Code show interactive meeting cards
(`DisplayMeetings`). Paperclip discards the MCP Apps metadata, so it
cannot show them.
**Proposed solution**
Paperclip implements the host side of MCP Apps end to end. It keeps the
tool's view declaration and records each call that has a view. It
renders the view under the tool row in a sandboxed iframe that uses the
MCP Apps postMessage protocol. Circleback becomes a native catalog
connection, so its `Display*` tools are available.
**Alternatives considered**
Composio's `circleback_mcp` toolkit has 11 read tools and no `Display*`
tools, and its meta-tools return plain JSON, so it cannot carry views.
The spec's double-iframe sandbox proxy on a separate origin needs a
second origin that Paperclip does not have. A single opaque-origin
`srcdoc` frame with an injected CSP gives the same isolation.
**Roadmap alignment**
This extends the existing Apps catalog and MCP gateway. It adds no new
integration substrate (decision D4 in doc/connections/README.md).
**Additional context**
The standard, the design and the testing plan are in
`doc/connections/MCP-APPS.md`.
## What Changed
- **Shared:** `packages/shared/src/mcp-apps.ts` adds MCP Apps types,
`_meta.ui` and resource parsing (current and legacy keys), and a CSP
builder that keeps only well-formed `https` origins.
- **DB:** migration `0471` (`online_compatible`) adds
`tool_catalog_entries.ui_meta` and a new `tool_mcp_app_views` table.
- **Server, catalog:** discovery keeps each tool's `_meta.ui`. It is not
part of the version hash, so a view never quarantines a reviewed tool.
- **Server, gateway:**
- Paperclip advertises the MCP Apps client capability when it
initializes a session.
- App-only tools (`visibility: ["app"]`) are hidden from agents.
- The result `_meta` is kept for the view and is not sent to the agent.
- Each successful call to a tool with a view records a view row.
Recording is best effort.
- The gateway's `initialize` now reports `paperclip/mcp-app-ui:
host_rendered`.
- The context protocol (`resources/read`) now uses the injected test
transport, like the other remote calls.
- **Server, routes:** `server/src/routes/mcp-apps.ts`
- `GET /api/issues/:issueId/mcp-app-views` lists a task's views. It
needs task read access.
- `GET /api/issues/:issueId/mcp-app-views/:viewId` returns the view
document, read live through the agent's connection.
- `POST /api/issues/:issueId/mcp-app-views/:viewId/tool-calls` runs the
view's `tools/call`. It needs `tools:use` and reaches only tools of the
same connection that allow `app`. It uses the connection test-call path,
so policy, Ask first and audit apply.
- **UI:** `ui/src/components/mcp-app/`
- The host protocol: `ui/initialize` with theme, style variables and
size limits; tool-input; tool-result; size-changed; `tools/call`;
`ui/open-link` (http and https only); teardown.
- An opaque-origin iframe (`allow-scripts`, never `allow-same-origin`)
with the CSP placed before the view's markup.
- The views of a run render with that run's reply, in the live turn and
in settled history. They are keyed by the reply's run, so history needs
no transcript.
- **Circleback:** a native catalog app (`mcp-oauth`, DCR, `user
user:write`) with:
- the official logo;
- a reviewed risk classifier (Share* tools are writes, Delete* tools are
destructive, Display* tools are reads);
- warnings in the sign-in step;
- `doc/connections/CIRCLEBACK.md`.
Search now prefers a native app over an aggregator listing named "X
MCP".
- **Storybook:** `Tasks/MCP App views` shows the tool row with its view,
the ready state (Refresh calls a tool), loading, view unavailable, and a
tool call that needs approval.
## Verification
- `pnpm exec vitest run packages/shared/src/mcp-apps.test.ts
ui/src/components/mcp-app/`: 19 passed.
- `pnpm exec vitest run server/src/__tests__/mcp-apps.test.ts`: 3
passed, on embedded Postgres. The test connects Circleback through the
catalog with DCR OAuth against a fake Circleback MCP server. It checks:
- `ui_meta` is stored;
- an app-only tool is hidden from the agent;
- an agent call records one view with arguments, structured content and
`_meta`;
- a call without a view records nothing;
- the routes serve the view's HTML and CSP;
- the view's tool call works and a model-only tool is refused;
- another task, another company and an agent actor are refused.
- Circleback connector: 13 focused files, 645 tests passed. These are
the app definitions, search, aggregator fallback, connection intents,
tool policy, `AppsConnect` and `Browse`.
- Typecheck passed for `@paperclipai/shared`, `@paperclipai/server` and
`@paperclipai/ui`. `node scripts/storybook-catalog.mjs` and the
migration compatibility check also passed.
- Storybook check in the browser:
- In both themes, the view runs the handshake, renders the meeting cards
from `structuredContent`, picks up the theme variables, and resizes to
its content.
- Refresh sends `tools/call` and re-renders the cards.
- **Live Circleback end to end** on a local instance, with a real
Circleback account:
- Connected through the catalog entry with Circleback's dynamic client
registration and OAuth.
- Discovery stored the real view declarations:
- `DisplayMeetings` uses `ui://circleback/meetings.v2.html`.
- `DisplayActionItems` and `DisplayCalendarEvents` also have views.
- `ListAssigneeOptions` and `SetCalendarEventRecording` are app-only
helpers with no view. They are hidden from agents.
- A `claude_local` agent called `DisplayMeetings` on two tasks.
- The real 3.5 MB Circleback view rendered the meeting cards with the
agent's reply.
- Clicking a card opened that meeting in Circleback through
`ui/open-link`.
- The live run found two bugs, both fixed in this PR, each with a
regression test:
- The 1 MB tool-result limit blocked the view document.
- Views were hidden inside the folded tool history.
## Risks
- **New migration:** a new table and a nullable column. It is
`online_compatible`.
- **Existing connections:** tools get `ui_meta` on the next catalog
refresh. Until then, no view shows and the tool row is unchanged.
- **Untrusted provider HTML:** the view runs as provider HTML in an
opaque-origin sandbox. It cannot reach Paperclip's session, DOM or API.
Its network access is limited to the origins its server declared. A
view's tool calls use the viewer's permissions and the normal policy and
Ask-first path.
- **Agent tool lists:** app-only tools are removed from them. No current
provider marks tools app-only, so no listed tool disappears.
- **Search:** the change that prefers a native app over an "X MCP"
aggregator listing affects every app.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and
subagents (one subagent authored the Circleback catalog connector).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
6a24b417c0 |
test(restart-chaos): add hosted mode for the release-swap proof on a real staging instance (#21153)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Hosted instances take an automatic release about every 40 minutes as an overlap swap: a new server starts beside the old one, both serve for a while, then the old one stops > - The restart-chaos harness (`tests/restart-chaos/`) proves release swaps only on servers it starts locally, with the release-swap cell RS1 (#21126) and the staging-mix workload (#21125) > - The zero-downtime finish line needs the same proof on a real hosted staging instance across real releases, with the same per-task grade > - This pull request adds a hosted mode: the harness starts no server and injects no fault; it runs the staging mix against one allowed hosted instance through its API, detects each real release from `/api/health`, and grades every started task with the RS1 grade module > - The benefit is one comparable number per hosted run (`tasks completed without human action / tasks started`) plus per-release continuity, with no database access and no risk to other instances ## Linked Issues or Issue Description Refs #21126 Refs #21125 Refs #21139 **Is your feature request related to a problem? Please describe.** The release-swap proof runs only against local servers. It cannot show what a real hosted instance does across its own automatic releases. **Describe the solution you'd like** A hosted mode in the harness. It takes one explicitly allowed staging or test host, signs in with an existing board token, creates labelled test agents and tasks, watches real releases through `/api/health`, grades every task with the RS1 grade, reports web and API continuity, and removes what it created. **Describe alternatives you've considered** Reading the hosted database directly. That needs production-grade database credentials and gives a second, different collector. The API path uses the same access a person has and reports the fields the API cannot supply as unavailable. ## What Changed - `hosted-guard.ts`: refuses to start unless `--hosted-base-url` is a bare `https://` origin and `--hosted-allow` names the same host. Only one stack under `.staging.paperclip.app` or `.test.paperclip.app` is accepted. The shared staging instance, production, the fleet admin and Cloud hosts are always refused. Every request is checked against that origin before it is sent. - `hosted-api.ts`: a bearer-token API client (token from `PAPERCLIP_HOSTED_TOKEN` or the CLI store, never printed) that logs every request for continuity, retries only reads, and a `/api/health` probe. - `hosted-release.ts`: pure release detection over health samples. It reports each release's overlap window (two process identities answering), failed and slow requests, time to the first answer from the new build, its shape (`overlap`, `stop_then_start`, `clean_switch`) and probe continuity (requests, failures, longest gap). - `hosted-readers.ts`, `hosted-grade.ts`: API-backed readers for runs, tasks, task views, pending questions and approvals, failure chains, next events and duplicated comments and subtasks. The grade calls `gradeReleaseSwapTask`, `summarizeGrades`, `swapMetrics` and `releaseSwapMarkdown` from `release-swap-grade.ts` (#21126), clicks Retry where the board offers it, and labels each failure. Fields the API cannot supply (queued wakes, the owning server process, runs lost while the old server lived, double executors, late-start and preparation probes) are listed as unavailable. - `hosted-setup.ts`: works in the signed-in user's company (Cloud refuses company creation). Creates one labelled project, one labelled agent per adapter and one labelled secret per provider key through the normal APIs. Cleanup pauses and removes the agents, cancels their live runs, hides their tasks, deletes the secrets and archives the project. It touches only names that carry the run label. - `hosted-workload.ts`: drives the staging-mix tasks from `staging-mix.ts` (#21125) at `--hosted-concurrency` (default 5), posts follow-ups during runs, and clicks an offered Retry once per failed run. - `hosted.ts`, `hosted-config.ts`, `hosted-main.ts`: the entry point, options, `--hosted-observe-only` (read-only release watch), `--hosted-cleanup-only`, and the run. `main.ts` and `supervisor.ts` are unchanged. - Hands-off grading (`hosted-handsoff.ts`, `hosted-workload.ts`): no Retry click during the wave or the settle period. Each task records when a Retry banner first showed and whether it cleared by itself. After the settle period the harness clicks Retry once on tasks that are not done and records whether it worked. `SUMMARY.md` starts with `<pass> / <started> tasks completed hands-off`, `Retry banner shown: <n>` and `Retry worked / clicked: <w> / <c>`. `--hosted-retry-during-wave` keeps the old behavior. - Assigned-work invariant: the grade reads `GET /api/companies/:id/assigned-work-invariant` once (from #21139) and reports the company's violation counts and the sampled violations and stalled slots of this run. A 404 on an older build is reported as not available. - `README.md`: a Hosted mode section with commands, the guard, sign-in, setup, detection, grade, unavailable fields and cleanup. ## Verification - `node server/node_modules/vitest/vitest.mjs run --config tests/restart-chaos/vitest.config.ts tests/restart-chaos/hosted-*.test.ts tests/restart-chaos/release-swap-grade.test.ts`: 52 tests pass (38 hosted tests). They cover the host guard (allowed host, mismatched allow flag, the shared staging instance, production, fleet admin, Cloud and look-alike hosts, a userinfo trick in a route), token lookup, release detection (overlap, stop-then-start, a restart on the same commit, unsettled release, continuity), the API readers, the end-to-end API grade with a Retry click and labels, the workload, and label-only cleanup. - `node server/node_modules/typescript/bin/tsc -p tests/restart-chaos/tsconfig.json`: clean. - Run without a token: the harness stops before any request and names the missing sign-in. Run against the shared staging host: refused by the guard. - `pnpm verify` asks for broader CI coverage (exit 2, README is a global CI input); required CI supplies it. - Known failures on master, outside this diff: `server/src/__tests__/storage-s3-multipart-throttle.test.ts` ("halves new parts in flight once when a parallel burst of parts is throttled", expected 3 to be 4) also fails on master and is being fixed separately. `packages/paperclip-runner` `src/drivers/acpx/native-distribution-integrity.test.ts` ("expires a real ACPX permission on native child death without replaying its mutation") is in the Runner package; this PR changes only `tests/restart-chaos/`. - Live smoke on one hosted staging instance (2 tasks): setup, workload, grade and cleanup ran end to end. It showed two harness defects, now fixed: an activity window that was too short for 90 s tool rounds, and agent removal that needed termination first. - Not run here: a hosted run across real releases (none reached the instance during this work). ## Risks - Low risk to the product: harness-only, no product code changes. - The run writes to one hosted instance. The guard, the per-request origin check and the label filter limit it to that instance and to objects it created. - The grade over the API cannot see queued wakes. A task whose only owner is a queued wake grades as having no owner. The report says so. - #21125 and #21126 are on master; this branch merges master and adds only hosted files and the README section. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits). Default context window; no extended thinking mode. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
ef6d347f79 |
fix(runner): keep reattached runners answering after a restart (#21161)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs execute in a remote Runner that the server controls over
the durable PRP channel
> - After a server restart, the next server reattaches to the live
Runner and sends recovery barriers (`runner.drain`, `session.snapshot`)
> - These barriers timed out while the Runner stayed connected, and each
same-run retry failed the same way
> - After three attempts the run stopped with
`native_session_retry_exhausted` and needed a manual Retry
> - This pull request finds the two causes and fixes both in the
controller and the transport
> - The benefit is that restart recovery finishes on its own, without a
manual Retry
## Linked Issues or Issue Description
**What happened?**
After a server restart, a run reattached to its live remote Runner. The
Runner authenticated. Then a recovery command got no recorded answer for
30 s or more, although the Runner stayed connected. Errors on staging:
- `PRP command session.snapshot timed out (runner connected; pending
ahead 0; last runner event seq 869 55s ago)`
- `PRP command runner.drain timed out (runner connected; pending ahead 1
[semantic_tool.result] oldest 170s ...)`
- `runnerd exited while waiting for runner.drain`
Each reattach failed the same way. After three attempts the run ended
with "Workspace repair required" (`native_session_retry_exhausted`).
Root causes, proven with the real Runner and the fake Codex harness:
1. The Runner answered in about 40 ms. At reconnect it also sends the
event backlog that it collected while the server was down. The
controller handles inbound frames of a connection in order, and it
commits each event (accounting write and checkpoint flush) before the
next frame. So the answer waited behind the whole backlog. With 400
events at 100 ms each, the answer was handled after 44 s, and the 30 s
barrier had already failed. The `semantic_tool.result` case is the same:
its answer waited behind the backlog, and the next command is sent only
after that answer is handled. The Runner code was not the cause.
2. When a Runner did not answer during close, the close path queued
`runner.drain` and `runner.suspend`, kept the Runner alive for
reattachment, and saved them in the journal. The next attempt delivered
that old `runner.suspend` to the reattached Runner, and the Runner
suspended and exited.
**Expected behavior**
A reattach to a live Runner completes recovery. A Runner answer that is
queued behind earlier frames is not a timeout. A reattach never delivers
a terminal command from an earlier attempt.
**Steps to reproduce**
1. Start a Codex run in a remote Runner with a turn that streams output.
2. Restart the server. Let the Runner collect a few hundred events while
it is disconnected.
3. Let the next server reattach while event commits are slow, as on a
loaded database.
4. See `session.snapshot timed out (runner connected; pending ahead 0)`.
5. Alternatively, let a close path leave `runner.drain` and
`runner.suspend` pending, then reattach. See `runnerd exited while
waiting for runner.drain`.
**Paperclip version or commit**
`master` at
|
||
|
|
a924d2e2c7 |
Persist portable Runner ownership through controller recovery (#21066)
## Thinking Path > - Paperclip delegates remote Runner execution to environment plugins. > - A plugin can move a session to another task allocation. > - The Core controller can lose an ownership reply or restart before the Runner starts. > - The source and target need a durable record of the same request. > - This change binds restart and attachment to that request and the original unlaunched task lease. ## Linked Issues or Issue Description **What existing behavior does this improve?** Recovery of plugin Runner sessions on replacement compute. **Current behavior** Core cannot replay a session ownership request from saved lease state during startup. **Proposed behavior** Record the operation on both leases and check the matched ownership receipt before attaching. Preserve the original pre-admission lease across controller restart. Managed environment configuration is in [#21065](https://github.com/paperclipai/paperclip/pull/21065). Related warm sandbox cleanup is in [#20065](https://github.com/paperclipai/paperclip/pull/20065). This layer preserves the same-sandbox fallback from [#21001](https://github.com/paperclipai/paperclip/pull/21001). Workspace placement follows in [#20743](https://github.com/paperclipai/paperclip/pull/20743). ## What Changed - Add the optional `sessionHandoffVersion` capability and the ownership task operation. - Save matching ownership claims and fence source checkpoint access. Replay a pending request on the original target. - Restore a portable session using the durable claim and the current credentials for the same account. Start a fresh conversation when the provider reports an unavailable checkpoint, or when Core's persisted compatibility decision selects a different session after transfer. The latter requires an unlaunched target; subsequent recovery must match the fresh task's saved identity. Both paths retain the workspace transfer and source fencing. - Reuse the original unlaunched task lease after hard or graceful controller restart. After a move has attached, validate the surviving Runner through the saved claim rather than requiring its logical lease ID to equal the new physical lease. - Keep older retained leases without a saved account identity on the existing resume path. - Apply the existing supported MCP-only compatibility rule to warm recovery and predecessor accounting. Persist the capability discovered at admission and reuse it after restart without repeating discovery. - Keep predecessor-retirement fallback scoped to the same physical sandbox. A refusal after a portable move preserves the session instead of requesting a replacement that the plugin driver cannot enforce. ## Verification - On `1eb527be27`, server typecheck (including Runner build and SDK dependencies) passed. - Independent final verification passed 168 tests across portable source recovery, session resume, controller recovery, and task authority; two existing tests were skipped. The full portable source integration suite passed 64 tests. - The new integration follows confirmed ownership through an incompatible instruction change, persisted fresh admission, target-only prepare/auth/submit without the source checkpoint, and reattachment after controller replacement. It also rejects unpersisted session selection, ambiguous target leases, and incorrect saved identities. - Before this final change, independent verification passed 1,667 tests across executor, session resume, recovery usage, remote warm recovery, runtime selection, and runtime execution; the environment lifecycle suite passed 208 tests. - Existing regressions cover previous-release account metadata, supported and unsupported MCP changes, provider/instruction incompatibility, and admission capability reuse without another discovery call. - On `51d7eec373`, all four S3 multipart throttle tests passed independently. CI exposed an existing timing-sensitive test twice; its 20 ms overlap assumption now uses synchronized retries and an event-loop yield. Production upload code is unchanged. Temporary concurrency caps of one and eight both fail the retained exact-four assertion. - Exact SDK source-pin checks passed for both stack heads. The upper #20743 code patch remains unchanged; its documentation preserves both layers' behavior. ## Risks - The handoff protocol is opt-in. Existing plugin leases without portable ownership metadata keep the previous release's resume path. Live attachment still requires end-to-end qualification. An unverified claim blocks startup. Git workspaces prepared on a portable allocation also need the workspace placement layer. ## Model Used - OpenAI Codex GPT-6 with GPT-6.1 Sol agents for implementation and independent review. ## Checklist - [x] Thinking path and model are included. - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] The issue is described using the enhancement template. - [x] Local checks passed and new behavior has tests. - [x] All Paperclip CI gates are green. - [x] Greptile is 5/5 with no open findings. --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
a4a0668a5c |
perf(attention): count the sidebar summary without the reads that only a card shows (#21159)
GET /attention/summary returns counts. To get them it ran the feed with a flag that skipped card bodies, and still read every blocked task through the full task list, every review two more times (one time with its description), the blocker and the blocked work of each card, and the spend of every budget policy. The summary keeps the collection path of the feed, so a card is counted by the rule that shows it. It no longer reads what only a card shows: - blocked work: the rows of the task list, in list order, with only the columns blocker attention reads (issueService.listBlockerAttentionRoots), classified by the function the full list calls - reviews: the review rows carry the card's task condition, so the summary does not read each review again - tasks of cards: without project and workspace - budget alerts: the open incidents (budgetService.openIncidents), not the spend of every policy - a fingerprint only for a card that has a dismissal; no queue membership and no "transferred by" read The review classification reads the 20 columns it uses, not the whole task. The request makes the same writes as before (decision queue seeds, retention states) through unchanged code. The feed, the task index and the task list return what they returned before. One summary request at 3,000 tasks: 133 -> 78 statements, 2,718 -> 842 KB fetched, 604 -> 144 KB of SQL text, 100 -> 45 MB allocated. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
cdefac664b |
test(restart-chaos): add the release swap cell RS1 with user-view grading (#21126)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart chaos harness (`tests/restart-chaos/`) proves that agent
tasks survive control-plane restarts
> - Staging takes a release about every 40 minutes as an overlap swap: a
new server starts beside the old one on the same database, both run for
8–9 minutes, then the old one gets SIGTERM
> - On 2026-11-12 staging failed or interrupted 577 of 1,000 runs across
these swaps, but the harness restarts one server at a time, grades each
task once and never repeats the swap
> - This pull request adds the release swap cell RS1: repeated overlap
swaps on one database, with the outgoing server starting new runs during
each overlap, and one end-of-wave grade that reads each task the way the
board shows it
> - The benefit is one number per wave, tasks completed without human
action over tasks started, that measures what staging users see
## Linked Issues or Issue Description
**Area**
Restart chaos harness (`tests/restart-chaos/`), test-only.
**Problem**
The harness did not reproduce the staging release shape. It had no
repeated swaps, no run started by the outgoing server during the
overlap, and no grade of every task at the end of a multi-swap wave.
**Proposed solution**
Add the `--release-swap` cell (RS1). It schedules K15 rolling redeploys
on a fixed clock, drives probe tasks through the outgoing server,
records per-swap evidence, and grades every task once after a settle
period.
**Why it matters**
The release cadence will not slow down. The harness must catch the
failures that the swap causes before staging does.
## What Changed
- `release-swap-schedule.ts`: the `--release-swap` options and the
`--release-swap-short` preset (3 swaps, 12 min apart, 510 s overlap)
(count 6, spacing 40 min, overlap 510 s, settle 15 min), their expansion
into a K15 overlap storm (adds `--k15-fresh-home --ephemeral-server` on
Daytona), the swap plan and the `ReleaseSwapRun` clock (swap k at k ×
spacing, never while a swap is unsettled; tasks launch until the last
swap settled).
- `release-swap-driver.ts`: the per-swap driver. It creates probe tasks
through the outgoing server's own port when the incoming server spawns,
when it enters startup recovery, and so that a run starts about 20 s
before the outgoing server's SIGTERM. It samples runs and their owning
controller PID every 2 s and records which server started each run. It
has a per-swap plug-in list; a plug-in can choose the checkout the
incoming server runs (for the later mixed-version variant).
- `release-swap-grade.ts`: the pure grade. One verdict per task (`pass`,
`needed_retry` with whether the click worked, `open_no_owner`,
`blocked_non_person`, `waiting_on_person`, `still_working`), the
per-swap numbers (failed runs, runs lost while the old server was alive,
time from SIGTERM to each surviving run's next event), duplicate runs
and duplicated side effects, the one-number summary, and one failure
label per failing task (`nothing pending`, `blocked for a non-person
reason`, `needed Retry` with the click result).
- `release-swap-finish.ts`: the end-of-wave collection. It waits for the
settle period, reads every task with a run from the API and the wave
database, clicks Retry where the board offers it (board UI when served,
else the same API call as the observer), waits again, and writes
`release-swap.json` and `release-swap-summary.md`.
- Small hooks in shared files: `supervisor.ts` calls `swapHook` at five
phases of the rolling redeploy and lets the hook choose the next server
root (the file stays at 1,000 lines); `config.ts`, `scheduler.ts`,
`types.ts` and `main.ts` wire the option, the plan, the clock, launches
during the overlap and the final grade.
- README section for the cell, its options and its grading.
- Unit tests for the scheduler, the option parsing, the grade
classifier, the per-swap numbers and the overlap evidence.
## Verification
- `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`
(58 files, 704 tests pass, after merging master).
- `node scripts/typecheck-test-projects.mjs tests/restart-chaos` passes.
- Local smokes through the wave queue (`--env local --adapters opencode
--workloads A1,T1 --release-swap --release-swap-count 2
--release-swap-spacing-minutes 5 --release-swap-overlap-seconds 90
--release-swap-settle-minutes 2 --concurrency 2 --tasks 40`):
- `w-20261112T151735-s7101` (first head): 10 / 16 tasks completed
without human action; 26 min wall. It showed that native runs had no
owner in the sampler (fixed here).
- `w-20261112T154840-s7102` (owner fix): 8 / 13 tasks completed without
human action; 24 min wall. The late probe's run was started by the
outgoing server 20.0 s before its SIGTERM.
- Both smokes caught the same product failure: runs the outgoing server
hands off at shutdown (`restartSuspendedRunIds`) are classified
`awaitingEvidence` by the incoming server's startup recovery, even while
the outgoing server is still alive, and are never settled (`I2_settled`
reconciliation violations). They stay `running` with no process, and the
runs queued behind them are never dispatched. This PR does not fix it
(harness only).
- `w-20261112T163456-s7103` (probe agents, drain): 28 / 29 completed;
the late probe started on the outgoing server 20.0 s and 20.2 s before
SIGTERM in both swaps.
- `w-20261112T171115-s7104`, with 40 preparation probes per swap
overloaded the probe agent (late probe dispatched by the new server 30 s
after SIGTERM); the cap is now 6 per swap.
- CI notes on earlier heads: `ci / e2e shard (1/8)` failed once in
`task-loading-frame` (UI, outside this diff; re-run). `ci / General
tests (server (11/12))` failed in
`src/__tests__/storage-s3-multipart-throttle.test.ts` ("halves new
parts…", expected 3 to be 4), outside this diff; it is a known master
failure being fixed separately and is not touched here.
- Not run: the Daytona smoke. No published runner image matches this
checkout's content ID; an image build for master is queued.
## Risks
- Low risk: test-only harness code. No product code changes.
- Waves without `--release-swap` keep their plans and behavior. The only
shared-path change is the K15 overlap redeploy, which now calls an
optional hook that is unset in other waves.
- The probe tasks and the sampling add a few API calls and one database
query every 2 s during each swap.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits). Extended reasoning on.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
b8207c4ea9 |
fix(runner): evict finished API snapshots instead of blocking reads at the company quota (#21133)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native agent runs read the Paperclip API through `call_api`. Large responses are saved as "API snapshot" assets so the agent can read them in pages. > - Each company has a 20 GiB snapshot quota. Before a large read, the server reserves 1 GiB of worst-case space against that quota. > - Snapshot assets were never removed. Reservations from interrupted captures (crash, restart, ambiguous upload) stayed at 1 GiB each until "an operator reconciles" them. Nothing did that. > - After a morning of high retry volume, the quota filled. Every large read then failed with `api_response_company_storage_limit`, and agents blocked their own tasks and asked an operator to free storage. > - This pull request makes company admission reclaim space by itself: it releases stale reservations of finished runs and evicts the oldest snapshots of finished runs, oldest first, only as much as the new read needs. > - The benefit is that a read never fails because old snapshots filled the quota, and no person must clear storage by hand. ## Linked Issues or Issue Description **What happened?** On staging, about 97 tasks of one agent had successful runs but ended in `blocked`. In each run, `GET` calls through `call_api` for large records (for example heartbeat runs and their events) failed with `api_response_company_storage_limit`: "Company API snapshot storage quota is exhausted. Read an existing asset or ask the operator to remove old snapshots or increase the quota." The agent then reported a task-wide blocker owned by "the instance storage operator". At least 15 tasks had a final comment such as "Blocked: company API snapshot storage is exhausted". The error message itself told the agent that an operator must act. **Expected behavior** A read must not fail because old snapshots filled the company quota. The server must reclaim the space automatically. Only snapshots that active runs still hold, or that users attached to issues, cases or logos, may stay. **Steps to reproduce** 1. Set `PAPERCLIP_RUNNER_API_COMPANY_CAPTURE_MAX_BYTES` to 2 GiB. 2. Store a 1 GiB `runner-api` snapshot asset from a finished run, or leave two 1 GiB unattached reservations from finished runs that are older than 20 minutes. 3. In a running native run, call `call_api` for a response larger than the inline limit. 4. Before this change, the call fails with `api_response_company_storage_limit`. After this change, the call succeeds and the oldest finished snapshot is evicted. **Paperclip version or commit** Staging deploy `704b3f4d63`. Fix is based on `origin/master` at `8bdd53c11f`. **Deployment mode** Hosted (staging cloud), native runtime with Daytona sandboxes. **Agent adapter(s) involved** Native Paperclip runner (Codex app server, `gpt-6.1-sol`). The defect is in the server-side `call_api` capture admission, so it affects every native adapter. ## What Changed - Added `server/src/services/native-runtime/runner-api-capture-retention.ts`. It measures company snapshot usage and reclaims space under the existing per-company advisory lock. - Reclaim step 1: release unattached reservations older than twice the capture deadline (20 minutes) whose run is missing or not `queued`, `running` or `scheduled_retry`. - Reclaim step 2: evict snapshot assets oldest first until the new reservation fits. A snapshot is eligible only when it is older than 1 hour, every run that captured it has finished (succeeded, interrupted, failed, cancelled or timed out), and no issue attachment, case attachment or company logo links to it. Each eviction writes an `asset.evicted` activity-log row with the asset id and the owning run id. - A capture upload that finishes after its reservation was reclaimed is rejected in the publish transaction. The same transaction records the object for deletion, and the bounded cleanup removes it. The upload has a 10-minute abort signal. - Failed deletes are logged per company and counted with the `runner_api_capture_delete_failed` metric. When cleanup keeps failing, the read error says so in plain words. - Eviction credits deletes that are already pending before it chooses more snapshots. Eviction is durable. In the same transaction that deletes an asset row, it inserts a reservation row with `pending_delete_object_key`, so the bytes stay counted. Admission then deletes the stored objects in pages, under a 30-second abort signal, and removes each row only after its delete succeeds. A failed or interrupted delete is retried at the next over-quota admission. No background poller is added. - Added the `pending_delete_object_key` column and an index on `runner_api_response_reservations.asset_id` (migration 0472). - `#reserveApiResponseCapture` now runs reclaim only when the quota would be exceeded, so reads below the quota do no extra work. After marking evictions it finishes the deletes and retries admission once. - When no space can be reclaimed, the error message now names what holds the space (runs still working, files attached to tasks, deletes in progress). It no longer tells the agent to ask an operator. - Replaced the test that asserted old snapshots block reads with tests for eviction, multi-page eviction, durable delete retry, a bounded stalled delete, stale-reservation release, and protection of active and attached snapshots. ## Verification - Red: the three new tests in `runner-api.integration.test.ts` fail on `origin/master` (two get `api_response_company_storage_limit`, one gets the old "ask the operator" message). - Green: `cd server && npx vitest run src/services/native-runtime/runner-api.integration.test.ts` passes 70/70 with the fix. - The multi-page test failed first with `api_response_company_storage_limit` because pending deletes ran in one 200-row batch. The paged delete loop fixed it. - `cd server && npx tsc --noEmit -p .` passes. ## Risks - An agent that reads a saved snapshot of a finished run after it was evicted gets a not-found response. It can read the API again. Eviction happens only under quota pressure, oldest first. - A released stale reservation can leave an unknown storage object from an ambiguous upload. These bytes were never linked to an asset, so they were already unreachable. This is the same storage the old code kept counted forever. - Eviction runs inside the company capture lock. The candidate query is bounded to pages of 200 rows and uses the new `asset_id` index. - Migration 0472 adds one nullable column (`ADD COLUMN IF NOT EXISTS`) and builds one index with `CREATE INDEX CONCURRENTLY IF NOT EXISTS`. It is listed in `MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS`. It is additive and safe for old and new servers during a rolling deploy. - During a storage outage, an over-quota read still fails until deletes succeed. The bytes stay counted, so the quota is never exceeded. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, browser reads of staging). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
0f3a6a867d |
fix(recovery): continue open agent tasks left with nothing running, pending or waiting on a person (#21139)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents work on tasks through runs, and recovery code starts the next run when a run stops > - On staging, many open agent tasks show "running" or "blocked", but no run, wake or retry is pending, and no person has to act > - Staging deploys about every 40 minutes. Each deploy stops some runs, and some tasks then stay idle until a person presses Retry > - The stranded sweep has many lanes. Some lanes get "no retry" back and skip the task with no record, and the sweep never looks at `blocked` tasks > - This pull request adds one invariant with one pure predicate, and one event-driven sweep that repairs a violation with one ordinary continuation wake > - The benefit is that no open agent task stays with nothing running, nothing pending and no one else's move ## Linked Issues or Issue Description Refs #21107 and #21046 (the pattern for leaving a run of another live server alone). Related open work: #21135 (fresh continuation from saved work at wake admission), #21124 (sweep calls the bounded scheduler for pre-provider legacy failures), #21127 (run-start hand-over). **What happened?** On a staging snapshot (2026-11-12, about 14:35Z), 1,030 open agent tasks had no live run. The new predicate, run read-only against the staging API, classifies 141 of them as violations: 40 offer a Retry only the system can act on, 38 are blocked with no recorded reason and no hold, 14 are blocked by an automatic recovery outcome with no hold left, 5 have nothing pending, and 44 are blocked under an active recovery hold (left to the hold's owner, see below). 25 more are blocked by recovery for more than 7 days. The other tasks wait on a person or on a pending intent: 542 chat threads, 122 agent-declared blocks, 102 credential waits, 51 dependencies, 28 questions. Nine Retry-started `in_progress` tasks show the main gap. A person pressed Retry, a deploy stopped the run during setup (`server_shutdown_interrupted`, no provider work), and the bounded scheduler then refused with `continuation_user_authorization_missing` ("could not revalidate the original user continuation") on every sweep. The sweep's escalation did not take effect either, so each task stayed `in_progress` with a Retry button and nothing pending, for up to 10 hours. **Expected behavior** Each open task assigned to an agent has a live run, a durable pending intent, or a named reason that is someone else's move. If it has none of these, Paperclip continues the task automatically. **Steps to reproduce** 1. Start a person's Retry on an `in_progress` task with a legacy agent. Stop the server during setup, so the run ends `interrupted` with `server_shutdown_interrupted`. 2. Let the bounded retry scheduler refuse to revalidate the person's continuation. 3. Run the stranded sweep. Nothing is queued. The task stays `in_progress` with nothing pending. 4. The same happens for a native predecessor (`enqueueStrandedIssueRecovery` returns `null` for every native run) and for every `blocked` task (the sweep reads only `todo`, `in_progress` and `in_review`). ## What Changed - `server/src/services/recovery/assigned-work-invariant.ts`: the invariant as one pure predicate. Facts go in. The verdict is `ok` with its clause (live run, pending intent, or one reason from a closed list of waits), `settling` with a re-check time, or `violation` with its reason. An unknown hold, block reason or person action is a violation. It also holds the per-task continuation allowance: at most 6 per hour (the server-restart lane's bound, the same cap as #21135) and, below the cap, a spacing of 2, 4, 8 and then at most 15 minutes. At the cap the next try waits until the oldest continuation leaves the hour, so a task that keeps needing continuation is retried at about 6 per hour, never left for hours. - `server/src/services/recovery/assigned-work-sweep.ts`: an indexed pre-filter, a batched fact loader, the repair and a read-only report. - A running run counts as live unless the shared owner proof (`owner-proof.ts`) proves its server gone. A held controller lease is live, so a run of the other server in a deploy overlap is never touched. A run whose owner is proven gone goes to heartbeat's existing lost-process settle path. Native running runs are always treated as live; the native settle watch judges their owners. - Native active paths match the stranded sweep's check, including #21108's exclusion of a native finalization that a person's Retry replaced. - A run that ended less than 90 seconds ago, or whose controller lease another server still holds, is `settling`. The sweep looks again at that deadline. - A pending question counts as the person's move, unless the person pressed Retry (or sent a message) after the question was asked. - The repair takes the task's row lock, checks again that nothing is pending, applies the allowance and writes a plain-language activity entry. This entry is the claim, so two servers make one wake. For a `blocked` task with no recovery hold, it also puts the task back to the status the block recorded. Then it enqueues one ordinary `issue_continuation_needed` wake. The wake has no retry link, so it does not use the task's failure allowance. With #21135, wake admission turns that wake into a fresh run from saved work when a hold would defer it. - A `blocked` task under an active recovery hold is reported as a violation but not repaired here: the hold's owner (#21135) re-continues it, so the two never continue one task twice. When #21124 escalates once on a permanent scheduler refusal, the task is blocked under a hold, so #21135 continues it; when no hold is left, this sweep continues it. Neither case leaves the task waiting on a person for a reason that is not the person's. - The claim re-runs the full predicate on a fresh read under the task's row lock, so a question, approval, blocker, monitor, wake or run that appeared after the first read stops the repair. - A lifted block is restored to the status the blocking entry recorded (`_previous.status`, `previousStatus` or `fromStatus`). A block that interrupted a review goes back to `in_review` with no agent wake; the review's own recovery wakes its participant. If admission refuses the wake, the block is put back with its original block time. - The settling window is keyed on three events only: the latest run ending (a held lease can extend it, but never past 5 minutes after the run ended), a never-run task's creation, and a blocker finishing. Routine touches (comments, reads, labels) never postpone a repair. A run cut by a deploy's shutdown is repaired about 90 seconds after it ended. - Each process starts at most 10 repairs per minute, newest-updated tasks first. The other tasks wait for a deadline. - Each burst of checks writes one structured log line (`assigned_work_invariant.pass`) with the counts by clause, the repairs and the deferrals. - `server/src/services/recovery/work.ts`, `issue-recovery-notifications.ts`, `delivery-work-notifications.ts`: a new `assigned-work-invariant` queue on the shared delivery-work coordinator. The existing issue-recovery notifications (run state changes, hold resolutions) and server start mark a company dirty. There is no interval polling. Deadlines exist only for a real obligation (settling grace, back-off, rate-limit deferral). - `server/src/services/recovery/service.ts`, `heartbeat.ts`: wire the sweep with the existing wake, live-execution and settle dependencies. - `server/src/routes/agents.ts`, `server/src/routes/openapi.ts`, `docs/api/agents.md`: `GET /api/companies/:companyId/assigned-work-invariant` (board only, company-scoped) and filtered by the caller's issue-privacy condition: private tasks the caller cannot read are absent from every count and sample returns the counts by clause, the violation count, the held subset, stale blocks and up to 20 samples. Use it after a deploy to check for 0 violations. Why there are two sweeps: the stranded sweep decides how a known failure resumes (bounded retries, disposition repair, review recovery, escalation). The invariant decides only that something is pending, and repairs with the ordinary continuation. It does not change the stranded sweep's lanes or the recovery-refusal paths. ## Verification - `server/src/services/recovery/assigned-work-invariant.test.ts` (45 tests): every clause, every wait reason, the unknown cases, settling, the allowance and its long back-off. - `server/src/__tests__/assigned-work-invariant.test.ts` (17 tests, embedded Postgres): a refused or later-declined continuation stays a violation and is retried under the cap; the real wake queue deferring behind a recovery hold; scheduling stops when a task is done, cancelled or reassigned to a person; running vs stopped or blocked session goals; the deploy-interrupted Retry shape and the native reconciliation shape (the stranded sweep does nothing, then exactly one wake); the nine Retry-started tasks whose scheduler refused revalidation, including one with a question asked before the Retry; a block after a legacy run's retries ran out with no hold left (`controller_lease_lost`); a block under an active hold left to its owner; two concurrent sweeps make one wake; another live server's run is not touched; an owner-gone run is settled; approval, question, dependency and declared-unblock tasks are not touched; an unknown block reason is repaired; the per-task cap and the per-process budget; a run that ended seconds ago; old-shaped rows (a board-owned stranded action from an older release, native `awaiting_runner_reattach` and `retryable_failure`, a running run saved without its controller process, a block older than 7 days); query cost at staging size. - Query cost at staging size (800 open tasks, 14,000 runs over 7 days): one suspect page 2 ms, one task check 165 ms (including the repair), a full 800-task report 386 ms. The suspect anti-join uses the run status index (no sequential scan on `heartbeat_runs`, 0.2 ms). - Route test in `agent-live-run-routes.test.ts`: a board operator reads its own company; another company and an agent get 403. - Existing production-wiring suites pass with the new queue: `issue-recovery-deadlines`, `issue-recovery-conversations`, `live-recovery-work`, `issue-recovery-work-lifecycle`, `recovery-deadline-acceptance`, `server-startup-feedback-export`, `heartbeat-held-comment-wake`, `delivery-work-coordinator`. - `node scripts/typecheck-layers.mjs` for the server source layers passes. ## Risks - On the first deploy, the sweep continues about 97 staging tasks over about 10 minutes (10 per minute). Each continuation is an ordinary run. - The stranded sweep can still move a task to `blocked` after its bounded retries. The task is then continued by the hold's owner or by this sweep, within the same cap and back-off. The task gets more activity entries than before. - On current master without #21135: when the wake reaches a task whose last run left a recovery hold, admission defers it (a skipped wake, no pending row). The activity entry then says the continuation did not start, the task stays a violation, and the sweep tries again with spacing (2, 4, 8, then 15 minutes at most; at most 6 per hour). It never loops faster than the cap and never moves the task to blocked for a person. A test with the real wake queue covers this. With #21135, the same wake becomes a fresh run from saved work. - The activity entry is written as an attempt and is published only after admission: as queued, or as not started. A wake that the queue declines later (for example behind the old run's cleanup) leaves the task a violation, and it is retried under the cap. - Chat-thread tasks are left out (the conversation clause), because their next move is a person's message. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, test runs, read-only staging API reads). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
49550814fd |
fix(runner-e2e): re-read a process listing that fails once during an owned stop (#21152)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Runner end-to-end harness stops the Paperclip server and its launcher through a process tree owner that reads `ps` > - The owner already waits out a briefly uncertain process group, but one failed `ps` listing ended the stop at once > - The stop then fell back to the direct child and always reported `Owned tree cleanup incomplete`, even when the launcher exited normally > - CI showed this as `Owned tree cleanup incomplete: Error: Could not inspect owned process identities` in `server-stop.test.ts` on 5 unrelated branches > - This pull request re-reads a failed listing a bounded number of times inside the same deadline, and names why `ps` failed > - The benefit is that one transient listing failure no longer fails a server stop, and a persistent failure is diagnosable ## Linked Issues or Issue Description **What happened?** `tests/runner-e2e/server-stop.test.ts > lets launcher cancellation join wrapper shutdown without signaling the server twice` (job "Runner support suite", project `runner-e2e-support`) fails intermittently with `Error: Owned tree cleanup incomplete: Error: Could not inspect owned process identities`, thrown from `tests/runner-e2e/process-tree-owner.ts:265`. It failed 6 times on 5 unrelated branches in 48 hours. **Steps to reproduce** 1. The failing inspection is `readProcessTable()` returning `null` (non-zero `ps` exit, `ps` timeout, or spawn error). The CI logs do not say which, because `ps` stderr was discarded. 2. Locally on macOS, 40 plain runs pass (the Linux `ps` failure does not occur on this host). 3. Inject the fault: make `readProcessTable()` return `null` for 15% of calls. 26 of 30 runs then fail with the exact CI error. **Paperclip version or commit** `master` at the base of this branch (CI on unrelated pull requests, 2026-11-10 to 2026-11-12). **Expected behavior** A failed listing grants no authority to signal anything. The stop reads the table again within its deadline, as it already does for an uncertain group, and it still falls back to the direct child when the listing keeps failing. **Actual behavior** `settleUncertainIdentity` only retried the uncertain-identity error. The first failed listing went straight to `incompleteTreeFallback`, which always throws. ## What Changed - `process-tree-owner.ts`: `settleUncertainIdentity` also re-reads after `Could not inspect owned process identities`, up to 3 attempts per step. It does not start a new read when less than one poll interval (100 ms) is left before the step deadline. No signal is sent between attempts. A persistent failure keeps the old fallback, so the direct child still gets its grace period. - `process-tree.ts`: `readProcessTable` takes an optional `onFailure(reason)` callback and keeps `ps` stderr. The owner adds the reason (exit code, signal, timeout, or stderr tail) to the inspection error. - `process-tree-owner.test.ts`: four regression tests. A listing that fails once during a stop no longer fails it. A listing that always fails still falls back to the direct child after 3 reads. No read starts at the deadline. A failed listing reports its reason. ## Verification - Red/green: both new tests fail on master. The first fails with the exact CI message `Owned tree cleanup incomplete: Error: Could not inspect owned process identities`. Both pass with this change. - Loop of the flaky test, 30 runs each, with a temporary local fault injection (`readProcessTable` returns `null` for 15% of calls, not committed): - Before: 4 pass, 26 fail (24 with `Owned tree cleanup incomplete: Error: Could not inspect owned process identities`). - After: 27 pass, 3 fail. In a 20-run recheck, the only failures (2) came from the test's own preliminary `await owner.observe()` at line 166, outside the stop path. The 15% injected rate is far above the real rate. - Without injection: 40 of 40 pass on master locally. - `npx vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/process-tree-owner.test.ts tests/runner-e2e/server-stop.test.ts`: 41 tests pass. - `tsc --noEmit -p tests/runner-e2e/tsconfig.json`: no errors in the changed files. ## Risks - Low risk. This is test harness code. Each step can now wait up to 2 extra 100 ms polls when `ps` fails. No new read starts within 100 ms of the deadline. As before this change, one `ps` read that is already running can still take up to its 5 s timeout. The exact cause of the Linux `ps` failure is not proven yet. The new error text will name it if it happens again. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
c5e4b46d3f |
fix(acpx): shut down the per-port ACPX runtime when its session closes (#21151)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The experimental Runner starts ACP providers through ACPX, and each runtime port owns one ACPX runtime > - When the native provider dies during a turn, the port retires the turn at once, but the turn stays queued inside the ACPX runtime > - `port.close()` closed only the session handle, so that queued turn wrote its final session record after close had returned > - Any caller that removes the state directory after close races that write; CI showed it as `ENOTEMPTY ... state/sessions` in an unrelated test on 5 branches > - This pull request makes close shut down the port's own ACPX runtime after the session closes, inside the existing close bound > - The benefit is that a closed port no longer writes into its state directory, and the flaky test is fixed at its cause ## Linked Issues or Issue Description **What happened?** `packages/paperclip-runner` test `native-distribution-integrity.test.ts > native ACPX execution closure > expires a real ACPX permission on native child death without replaying its mutation` fails intermittently with `Error: ENOTEMPTY: directory not empty, rmdir '/tmp/native-acpx-test-XXXX/state/sessions'`. It failed 5 times on 5 unrelated branches in 48 hours. **Steps to reproduce** 1. Run the test 30 times while two CPU-bound shell loops run. 2. 2 of 30 runs fail with ENOTEMPTY in `afterEach`. 3. Instrumenting the session store shows a `saveSessionOwner` write from `finalizeRuntimeTurn` that starts after `port.close()` and `afterEach` have begun. **Paperclip version or commit** `master` at the base of this branch (CI on unrelated pull requests, 2026-11-10 to 2026-11-12). **Expected behavior** After `port.close()` resolves, the ACPX runtime owned by that port does no more work and writes no more state. **Actual behavior** The turn that native provider death retired is still queued in the ACPX runtime manager. It finalizes after close and writes the session file while the caller removes the directory. ## What Changed - `acpx-runtime-adapter.ts`: after the final successful close, the port calls `runtime.shutdown()` within `runtimeCloseTimeoutMs`. ACPX shutdown waits for its pending turn tasks, so the abandoned turn finishes its writes first. - A close that succeeds late, after its bound, also starts the shutdown. A concurrent or repeated `close()` waits for the same shutdown and gets its failure too. A shutdown failure is reported in the existing `ACPX runtime and provider cleanup failed` error. The shutdown attempt stays in the retained cleanup owners. - `acpx-runtime-adapter.test.ts`: regression tests. Close does not resolve before runtime shutdown settles. A shutdown failure is reported to every closer. A late close success starts the shutdown. ## Verification - Red/green: both new tests fail on master (`expected "vi.fn()" to be called once, but got 0 times`; `promise resolved "undefined" instead of rejecting`) and pass with this change. - Loop of the flaky test, 30 runs each, with 2 extra CPU-bound shell loops on an already loaded 18-core Mac (load average about 30): - Before: 28 pass, 2 fail (both `ENOTEMPTY ... state/sessions`). - After: 30 pass, 0 fail. - `npx vitest run src/drivers/acpx` in `packages/paperclip-runner`: 92 files, 1743 tests pass. - `tsc --noEmit -p packages/paperclip-runner`: clean. ## Risks - Low risk. The runtime is created per port and is not used after a successful final close. Close can now take up to one more close bound if the runtime's pending work is slow. A late reconciliation close never runs after this point, because shutdown only starts when the port is finally closed. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
eca1472bae |
test(e2e): stop Cloud auth health stub from reading a disposed response (#21150)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The e2e suite proves that a Cloud tenant returns to its task after Cloud sign-in > - `tests/e2e/cloud-auth.spec.ts` "Cloud recovery preserves the task" failed in CI e2e shards on at least 6 unrelated PR branches in 48 hours > - The error was `apiResponse.json: Response has been disposed`, which blocks unrelated PRs and hides real failures > - The `/api/health` route stub called `route.fetch()` and `response.json()` for each health request. A health request that started just before the test ended kept that handler in flight. Context teardown then disposed the fetched response, and the handler failed. > - This pull request reads the real health payload one time through the test-scoped `request` fixture and fulfills each intercepted health request from that snapshot > - The benefit is that no route handler does network work that can outlive the browser context, so the test cannot fail at teardown ## Linked Issues or Issue Description **What happened?** `Cloud recovery preserves the task (https://my.paperclip.app, task|auth)` and `(https://my-staging.paperclip.app, task)` failed intermittently in CI with `Error: apiResponse.json: Response has been disposed` at the `/api/health` route handler. **Steps to reproduce** 1. Route `/api/health` with a handler that awaits `route.fetch()` and then `response.json()`. 2. Start a `/api/health` fetch in the page just before the test ends. 3. The browser context closes while the handler is in flight. Playwright disposes the fetched response, and the handler throws. **Expected behavior** The test passes when the UI starts a health request near the end of the test. **Root cause** Playwright stores `route.fetch()` response bodies on the browser context request context. Context close clears them (`fetchResponses.clear()`), so `response.json()` (or `route.fetch()` itself, with "Test ended") fails in any handler that is still running. The UI fetches health from several components on mount and after reload, so a request is sometimes in flight at teardown. ## What Changed - `tests/e2e/cloud-auth.spec.ts`: get `/api/health` one time through the test `request` fixture, add the Cloud overrides, and fulfill intercepted health requests from that snapshot. The route handler no longer calls `route.fetch()`. - The snapshot also sets `status: "ok"`. A snapshot taken during startup recovery reports `starting`, which would keep the Cloud gate on the starting screen for the whole test. ## Verification - Before the fix: `PAPERCLIP_E2E_PORT=3293 npx playwright test --config tests/e2e/playwright.config.ts tests/e2e/cloud-auth.spec.ts -g "Cloud recovery" --repeat-each=30` gave 119 passed, 1 failed (`route.fetch: Test ended ... while running route callback` in the `/api/health` handler, the same teardown race). - Deterministic red/green probe (temporary spec, not committed): the page fires five `/api/health` fetches as the last step of the test. With the old handler: 15/15 failed with the exact CI error `apiResponse.json: Response has been disposed`. With the snapshot handler: 15/15 passed. - After the fix: `PAPERCLIP_E2E_PORT=3293 npx playwright test --config tests/e2e/playwright.config.ts tests/e2e/cloud-auth.spec.ts --repeat-each=30` gave 150 passed, 0 failed (120 Cloud recovery runs plus 30 self-hosted runs). After the `status: "ok"` follow-up: `--repeat-each=10` gave 50 passed, 0 failed. ## Risks - Low risk. Test-only change. The health payload is now a snapshot taken at the start of the test, not a live read. The test only depends on the overridden Cloud fields, and the other health fields do not change during one test. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), running in Claude Code as an agent with tool use (shell, file edit, Playwright runs). Standard context window, extended reasoning enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>canary/v2026.1112.0-canary.14 |
||
|
|
90ae3c4337 |
fix(native): process a superseded export's late stop receipt and stop its sandbox when none arrives (#21143)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native run can be retired as superseded while its workspace export fails. Its files stay unsaved, and the run stays the writer of the task's workspace. > - #21136 records those files for the automatic workspace recovery. That record needs the sandbox's stop-and-retain receipt. > - The native run projection worker checked for that receipt for one hour after the run ended. After that, it dropped the run. Nothing woke it when the receipt arrived later. > - If the receipt came late or never, the run kept the task's workspace fenced. Every queued run on the task waited and then failed setup. Nobody owned the problem. > - This pull request makes the worker process a late receipt, stop and keep the sandbox itself when no receipt arrives, and never drop the run silently. > - The benefit is that the task's queue moves again by itself, and two writers never use one workspace. ## Linked Issues or Issue Description Refs #21136 (review thread on `native-superseded-workspace-export.ts:84`) **What happened?** `preserveSupersededWorkspaceExport` returned `recheckAt: null` when the sandbox stop receipt had not arrived within `finishedAt + 1h`. The projection worker then deleted its entry. A later stop receipt (`settleStopOnlyCleanup`) woke only the workspace recovery queue, not the projection queue. `preserveNativeFailedWorkspace` requires `hasConfirmedSandboxStopAndRetain`, so no hold was recorded. The run stayed the dirty writer, and queued runs kept failing setup. **Expected behavior** A late receipt is processed at once. When no receipt arrives, Paperclip stops and keeps the sandbox itself and waits for the provider's confirmation. The workspace stays held while the sandbox may still run. When the provider confirms that the sandbox is gone, the task continues from its last save. When the stop stays unconfirmed for hours, the task shows one plain notice, and the check continues. **Steps to reproduce** 1. Retire a native run as superseded after its workspace export fails (the #21136 shape). 2. Let the run's sandbox stop not settle (for example, the server that owned the release is replaced). 3. Wait more than one hour after the run ended. 4. See that the projection worker drops the run, and that a later stop receipt records nothing. The workspace stays held. ## What Changed - `sandbox-stop-and-retain.ts`: a confirmed stop-only receipt now also notifies `DELIVERY_QUEUES.nativeRunProjection` (scoped to the company and run). New `prepareUnconfirmedSourceSandboxStop` puts a stop-and-retain intent on the source lease from its current status. It skips a destroyed sandbox, a stop already in progress, and a cleanup in flight. - `native-superseded-workspace-export.ts`: - Within one hour of the run ending, the worker checks every minute (unchanged). - After that hour, it requests the stop-and-retain itself. The existing pending-cleanup sweep dispatches the stop and retries it until the provider confirms. The worker checks again every 15 minutes, and the receipt wakes it at once. The workspace stays held. - A sandbox that another lease holds is never stopped. - When the provider confirms that the sandbox was destroyed, the files cannot be saved. In one transaction (task, run, then workspace locks), the worker sets aside only that run's workspace epoch, wakes queued-run dispatch, and records `issue.previous_run_files_lost` on the task. - When the stop is still unconfirmed six hours after the run ended (or the sandbox is held by another lease), the task gets one `issue.previous_run_files_waiting` activity entry. The worker keeps checking. It never drops the run. - A stop-only dispatch now settles when the provider confirms that the exact sandbox is already gone. Before, the runtime accepted only a `stopped` reply and threw on `destroyed`, so a gone sandbox kept the stop pending forever. - `sandbox-stop-and-retain.ts`: new `acceptedStopOnlyReceipt` accepts `stopped`, or `destroyed` for a stop-and-retain intent (a native workspace export resume still needs `stopped`). `settleStopOnlyCleanup` stores a `destroyed` receipt as `released` / `success` / `workspace_source_unavailable`, removes the stop intent, never writes a retain receipt, and wakes the projection and workspace recovery queues. - `environment-runtime.ts`: the plugin `environmentStopLease` and built-in `stopLease` checks use `acceptedStopOnlyReceipt`. - `workspace-recovery.ts`: when its own stop settles as `destroyed`, the source is unavailable (the existing alternative-strategy path) instead of a binding failure. - Daytona plugin: stop of a sandbox that Daytona reports as not found returns `destroyed` (destroy already did this) instead of throwing. boxd already returns `destroyed`. Other sandbox plugins have no stop-only method, so nothing changes for them. - New shared `SANDBOX_ALLOCATION_HOLDER_STATUSES` (`active`, `retained`, `pending_cleanup`). The stop-only cleanup ownership check and the worker's "held by another lease" guard both use it, so a retained warm sandbox is never stopped and the two checks cannot drift. - `ui/src/lib/activity-format.ts`: plain labels for the two task activity entries. They do not ask a person to fix anything. - `doc/execution-semantics.md`: document the rule. - Not changed: the shared owner prover still refuses to treat a stopped instance on another host as proof that its child processes stopped. ## Verification - `server/src/services/superseded-workspace-export-recovery.test.ts` (4 new tests, red/green): - "checks again every minute while the run's own release may still stop its sandbox": the recheck branch. Nothing is recorded or stopped, and the workspace keeps its writer. - "stops and keeps the sandbox itself when no stop receipt arrived, and records the files once the late receipt lands": after the hour, the lease gets a stop-only intent. A second pass changes nothing. The late receipt wakes the projection queue. The next check records the hold, and the system recovery frees the workspace. Fails on master (`expected { …(19) } to match object { status: 'pending_cleanup' }`). - "names the unconfirmed stop on the task once, and keeps checking, when the provider never confirms it": the expiry branch. One notice, the check continues, no hold, and the workspace stays held. Fails on master. - "continues from the last save when the provider confirms the sandbox is gone": epoch 1 is set aside, the workspace is free, the task says so, and the result is not applied. Fails on master. - Review-round tests (all 5 fail on the first head `a208bfa58f`, pass now): - `environment-runtime.test.ts` "settles a plugin stop whose provider reports the sandbox gone (boxd, Daytona not found) through the cleanup sweep": provider dispatch through `releaseRunLeases` and `sweepPendingCleanupLeases`. The lease settles `released` with a `destroyed` receipt and no retain receipt. - `environment-runtime.test.ts` "settles a built-in stop whose provider reports the sandbox gone, without claiming retained files": the built-in `stopLease` path. No destroy call. - `superseded-workspace-export-recovery.test.ts` "continues from the last save when Paperclip's own stop finds the sandbox already gone": the worker's stop request gets a `destroyed` reply, the projection queue wakes, and the workspace is freed at epoch 2. - `superseded-workspace-export-recovery.test.ts` "never stops a sandbox a retained warm lease still holds, says so once and keeps checking": the source lease is unchanged, a recheck is returned, and `sandbox_in_use` is recorded. - Daytona `plugin.test.ts` "explicit stop confirms a sandbox Daytona no longer has as destroyed". - `environment-runtime.test.ts` (207) and Daytona `plugin.test.ts` (421) pass. - Related suites pass: `native-run-projection-work`, `retained-native-session-cleanup-work`, `workspace-recovery-stop-proof`, `native-failed-workspace-recovery`, `legacy-workspace-restore-recovery`, `environment-retained-orphan-sandbox`, `workspace-recovery-retry-copyback`, `workspace-recovery-fallback` (398 tests), and `ui/src/lib/activity-format.test.ts`. - `tsc --noEmit` for `server`. ## Risks - Low to medium risk. The new behavior acts only on an ended superseded run that failed in its workspace export, still holds a dirty workspace, and has no retained-file hold. - The worker now stops an ended run's sandbox one hour after the run ended if no stop receipt arrived. The stop uses the existing stop-only contract and keeps the sandbox. A sandbox held by another lease is never stopped. - A destroyed sandbox's unsaved files are not recoverable. The task continues from its last save and says so. - If a provider never confirms a stop, the workspace stays held (by design: no second writer). The task shows one notice, and Paperclip keeps retrying. - Shared cleanup code now treats a confirmed `destroyed` stop-only reply as settled. Every caller was checked: the cleanup sweep and run release count it as settled; the workspace recovery treats the source as unavailable; the fallback treats it as stopped (nothing can run there). No caller reads it as retained files, because no retain receipt is written. A native workspace export resume still requires `stopped`. - Each extra delivery notification per settled stop-only cleanup is cheap. The projection worker's scoped load finds nothing for runs that do not match. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (code search, shell, test execution) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
04082a8fd6 |
test(heartbeat): check preparation span parents only in the attempt trace (#21149)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The heartbeat service prepares each run in named phases and traces
each phase as an OpenTelemetry span under the execution attempt span
> - The test `heartbeat-preparation-gap-spans.test.ts` checks that each
named preparation span has the execution attempt span as its parent
> - The test failed in CI 5 times in 48 hours on unrelated PR branches,
always for `heartbeat.prepare.chat_retry_admission`
> - The cause is in the test: the native run trace also replays each
recorded preparation phase of 50 ms or longer as a historical span with
the same name, under `task.prepare` in a separate trace. When the
machine is slow, `chat_retry_admission` takes more than 50 ms, and the
test also checks the parent of the replayed copy
> - This pull request limits the span checks to the trace of the live
execution attempt
> - The benefit is that the test is stable and still proves that every
live preparation span is a child of the execution attempt
## Linked Issues or Issue Description
**What happened?**
`server/src/__tests__/heartbeat-preparation-gap-spans.test.ts` failed
intermittently in CI with `AssertionError:
heartbeat.prepare.chat_retry_admission: expected '<span id>' to be
'<attempt span id>'`. It failed 5 times in 48 hours on 5 unrelated PR
branches.
**Expected behavior**
The test passes on every run. It checks only the live preparation spans
of the execution attempt.
**Steps to reproduce**
Run the test 4 times in parallel on a loaded machine: `cd server && npx
vitest run src/__tests__/heartbeat-preparation-gap-spans.test.ts`. A
span dump of a failed run shows two
`heartbeat.prepare.chat_retry_admission` spans. One is the live span
under `task.execution_attempt` (correct). The other is the historical
replay under `task.prepare` in the native run trace, made because the
phase took 72 ms (the record limit is 50 ms).
**Paperclip version or commit**
master at
|
||
|
|
3fb35fcf13 |
fix(native): restart an interrupted bootstrap whose runner connected before the deploy (#21120)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Agent turns run in a remote Runner (`paperclip-runnerd`) in a
Daytona sandbox. A deploy stops the server and a new server recovers
each run.
> - When the server stops during a run's bootstrap, recovery restarts
that bootstrap on the same run (`bootstrap_incomplete`). The claim is
only granted when the controller checkpoint in Postgres proves that no
provider turn was sent.
> - Host preparation then reads the sandbox Runner state. A Runner that
already authenticated is `ready`, not `suspended`. The restart is
allowed only with a stop proof for that Runner.
> - The stop proof was taken in two places only: from a restored local
controller root, or for a Runner that never connected. A new server
after a deploy can have neither, so the run failed with
`runner_harness_state_mismatch: resumed_not_suspended` and needed a
manual Retry.
> - This pull request lets host preparation take the same Postgres
checkpoint proof that granted the claim, stop that exact Runner, and
restart the bootstrap.
> - The benefit is that a deploy during a bootstrap no longer fails the
run. No provider turn can be replayed, because the proof shows that none
was sent.
## Linked Issues or Issue Description
**What happened?**
On staging, three Codex runs on Daytona failed with
`native_session_interrupted` and `runner_ingress_unavailable: runnerd
process failed: runner_harness_state_mismatch: resumed_not_suspended`
(12:48Z twice and 13:19Z on 2026-11-12). In each run the Runner
authenticated 1 to 4 seconds before a deploy stopped the server
("controller authority detached for restart; durable runner left
alive"). The new server chose "Restarting an incomplete native runner
bootstrap on the same heartbeat run". Its host preparation found the
sandbox Runner `ready`, had no stop proof, and failed the run. The run
log has no "Stopped the interrupted runner" line, so the earlier stop
step did not run on the new server.
**Expected behavior**
The run restarts its bootstrap on the same sandbox and continues without
a manual Retry. A provider turn that may have run is never replayed.
**Steps to reproduce**
1. Start a Codex run on a Daytona sandbox.
2. Deploy the server after the Runner authenticates and before the first
turn starts.
3. Let the new server recover the run with a `bootstrap_incomplete`
claim on a server that has no local controller root for the run.
4. Host preparation fails with `runner_harness_state_mismatch:
resumed_not_suspended`.
**Paperclip version or commit**
Staging at
|
||
|
|
0bac3510db |
test(restart-chaos): add the staging-mix workload for release-swap waves (#21125)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness (`tests/restart-chaos/`) restarts the control plane during real agent tasks and grades each task > - On staging (2026-11-12), 577 of 1,000 runs failed or were interrupted across overlap deploys every 40 minutes > - The harness missed this: its waves run a few short tasks, while staging runs about 30 agents at once with long tool turns, subtasks, comments and Retries during each deploy > - This pull request adds a staging-like workload (`--workload staging-mix`) that runs under any existing restart cell today, and under the coming release-swap cell > - The benefit is one honest number per wave (tasks completed without human action / tasks started) under a load shaped like staging ## Linked Issues or Issue Description **What existing behavior does this improve?** The restart chaos harness workloads. Every current workload is a few short tasks, so a wave never has 30 runs, a queue, long mid-tool turns, subtasks, or comments and Retries during a deploy overlap. **Subsystem affected** Test harness only: `tests/restart-chaos/`. No product code changes. **Current behavior** A wave plans a fixed list of short tasks with one fault each. The harness holds every board action while a fault is in progress, so no comment or Retry ever lands inside a deploy overlap. **Proposed behavior** `--workload staging-mix` plans a mix of OpenCode, Codex and Claude tasks (default 60/25/15). Tasks keep arriving for the whole wave. Each task runs 15–45 minutes of foreground tool rounds, parents create subtasks, and follow-up comments and Retries wait for an overlap window. Each task has a deliverable that the harness checks from the result. **Reason and benefit** Release-swap waves need a workload shaped like staging. A pass then means the work was done, not only that a status says `done`. ## What Changed - `staging-mix.ts`: the options (`--workload staging-mix`, `--mix-*`), the seeded plan (interleaved adapter ratio, arrivals over the whole wave, one fault slot per `--mix-fault-spacing-minutes`), the SM task (rounds of a hashing shell command, subtasks, `DONE-<nonce>` comment) and its oracle. - The oracle recomputes each round's chained POSIX `cksum` hash. It fails a missing, short or wrong round, a repeated round, comment or subtask (duplicated side effects), a follow-up that did not reach the ledger, and a parent that did not read its subtasks' results. - The follow-up and Retry step waits for an overlap window, for at most `--mix-hold-minutes`. A swap driver registers the window with `setOverlapProbe`. Until then, the window is a fault in progress while the server still answers (K15 overlap). Retries go through the observer's existing click path, so R3 still counts them. - `staging-mix-run.ts`: the arrival gate, the end of arrivals, and the resource guard. The guard refuses `--mix-runs` above the local cap, samples the harness process tree's memory and CPU and the host's available memory every 30 s, and holds launches while memory is low. - The three agents' `maxConcurrentRuns` split the runs by the ratio. The harness keeps runs plus `--mix-queue` tasks in flight, so a queue forms in Paperclip. - Small hooks in `config.ts`, `types.ts`, `main.ts`, `observer.ts`, `seed.ts`, `workloads.ts` and `journeys.ts`. `supervisor.ts` is unchanged. - README section "Staging mix", and `staging-mix.test.ts` (18 unit tests). ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 54 files, 657 tests pass. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: passes. - The unit tests compare the TypeScript `cksum` with the system `cksum`, and run the round command in a real shell. - Live smoke and sizing results: see the comment on this PR. ## Risks - Low risk. The change is in the test harness only. Existing waves do not set `--workload`, so their plans and grades are unchanged. - A 30-run Daytona wave needs Daytona vCPU headroom above the default 40 vCPU budget. The org is shared with the live fleet, so `RESTART_CHAOS_DAYTONA_VCPU_BUDGET` must be raised only to the real headroom. - Fault slots fire relative to each carrier task's real launch, so a slot can drift when launches wait for memory or the in-flight cap. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, file edits). Extended reasoning was available. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
792c17aac7 |
fix(ui): freeze the clock in the persisted PR-read cache test (#21147)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The board UI saves approved reads to local storage. A reload can
then show them at once.
> - The test `bounds saved PR reads and retains their original age
through later saves` checks the limit of 100 saved PR reads.
> - The test reads `Date.now()` one time and uses that value as the
cache clock. Then `seed()` writes the session and company list with the
real clock.
> - If the real clock moves 1 ms or more between these two steps, the
saved session has a time in the future. The cache correctly rejects that
session, so it restores no data.
> - This failed CI 5 times in 48 hours on 5 unrelated pull requests.
Each failure was `expected [] to have a length of 100 but got +0`.
> - This pull request freezes the clock in that test, as the sibling
tests in the same file already do.
> - The benefit is a deterministic test, and no more unrelated red CI
runs.
## Linked Issues or Issue Description
**What happened?**
The UI test `ui/src/lib/persisted-query-cache.test.ts > persisted read
cache > bounds saved PR reads and retains their original age through
later saves` failed in CI 5 times in 48 hours on 5 unrelated branches.
The error was `AssertionError: expected [] to have a length of 100 but
got +0` at line 82.
**Expected behavior**
The test passes on every run, independent of machine load.
**Steps to reproduce**
1. Start two CPU busy loops (`node -e 'while(true){}'`).
2. Run `pnpm --filter @paperclipai/ui exec vitest run
src/lib/persisted-query-cache.test.ts -t "bounds saved PR reads"` 30
times, two at a time.
3. Before this change: 0 pass, 30 fail, all at line 82.
Root cause: the test injects a fixed clock (`now`) into the cache, but
`seed()` calls `setQueryData` without `updatedAt`. TanStack Query then
uses the real `Date.now()`. When the real clock passes the fixed clock,
`fresh()` sees a session read from the future. The flush then does not
save the session, and the restore aborts. The product code is correct.
The test clock was not consistent.
## What Changed
- Call `vi.useFakeTimers()` at the start of `bounds saved PR reads and
retains their original age through later saves`. `Date.now()` is now
frozen, so the seeded reads and the injected clock agree. The other
clock-driven tests in this file already do this.
- Call `vi.useFakeTimers()` in `discards snapshots after %s changes`. It
had the same race. There, the race did not fail the test, but it let the
test pass without the discard path under test.
## Verification
- Deterministic red/green: I added a real 5 ms sleep (`Atomics.wait`)
after `let now = Date.now()` in a temporary copy of the test. Without
the fix: `expected [] to have a length of 100 but got +0`. With the fix:
pass. I removed the temporary copies.
- Loop under load (two CPU busy loops, two parallel vitest processes, 30
runs each):
- Before: 0 pass, 30 fail (all at `persisted-query-cache.test.ts:82`).
- After: 30 pass, 0 fail.
- Full file: `pnpm --filter @paperclipai/ui exec vitest run
src/lib/persisted-query-cache.test.ts`: 54 passed.
## Risks
- Low risk. The change is test-only. Fake timers are reset in the
existing `afterEach` (`vi.useRealTimers()`).
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edit) and reasoning.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
dfc510b887 |
fix(server): hand new run starts to the replacement server during a deploy overlap (#21127)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Paperclip Cloud replaces the server process on each release. The old server keeps serving until the new one passes health checks and takes the traffic. > - During that overlap the old server keeps starting new runs right up to SIGTERM, although the new server can already start them. > - SIGTERM then cuts off exactly those runs while they still prepare. In one audited swap the old server started 30 runs after the new server booted, and 23 of them ended `process_lost`. > - Admission must never pause, so the fix must hand over who starts new runs, not stop admission. > - This pull request makes the new server record when it starts runs. The old server then leaves new run starts to it, and starts them again if the new server stops or dies. > - The benefit is that a deploy no longer starts runs on a server that is about to stop, and the company always has one server that starts runs. ## Linked Issues or Issue Description Refs #20852 (preserves preparing and launching turns that a deploy interrupts; this PR reduces how many such turns the old server starts). Related: #21046, #21107 (shared controller-instance owner proof), #20986 (queued-run dispatch signal). **What existing behavior does this improve?** The start of new runs while two servers overlap during a Cloud deploy. **Subsystem affected** Server run admission (`server/src/services/heartbeat/queue.ts`), the controller instance table, and startup and shutdown in `server/src/index.ts`. **Current behavior** The old server starts queued runs until it receives SIGTERM. Runs that it starts in the last minutes of the overlap are interrupted while they prepare. Nothing tells the old server that its replacement already starts runs: `controller_instances.epoch` is written at process start, and that happens minutes before the new server can start runs. **Proposed behavior** The new server records `dispatching_at` and a short `dispatch_lease_expires_at` on its `controller_instances` row right after its startup queued-run pass. The old server reads this through an in-memory flag at run admission and leaves new run starts to the new server. Runs it already started continue and are handed over at SIGTERM as today. If the new server stops, drains or dies, the old server starts runs again within seconds. **Reason and benefit** Fewer runs are cut off by each swap. There is never a moment without a server that starts runs. **Breaking changes** None. The migration adds two nullable columns. Releases without this code ignore the columns, and their rows never carry the marker, so a mixed-version overlap behaves as today. ## What Changed - `controller_instances`: new nullable columns `dispatching_at` and `dispatch_lease_expires_at` (migration `0471_controller_instances_dispatch_marker`, two `ADD COLUMN` statements, no lock beyond the catalog update). - New `server/src/services/run-start-handover.ts`. It is one worker on the shared delivery coordinator (`DELIVERY_QUEUES.runStartHandover`). Its renewal and recheck are coordinator deadlines, and a drain change or expiry wakes it. - Replacement side: `begin()` records the intent and writes the marker (it creates the boot's row if the row is missing). If the server drains at that moment, the worker writes the marker when the drain ends. The worker then publishes a `run-start-handover` hint and renews the 20 s lease every 5 s. It renews only while an older boot that is not stopped or fenced may still serve, and for at most 30 min. All marker writes run in one serialized chain. `release()` at shutdown lands after any renewal already in flight. A clear that fails is retried, and shutdown waits for the release. While the server cannot dispatch (drain, quiesce, admission closed), the worker clears the lease. - Old-server side: `isYielding()` is an in-memory read. The hint refreshes it, and so does one read that a dispatch pass waits for (at most 1 s) when the last read is more than 2 s old. A read that gets no answer within 1 s never keeps the server yielding. The server admits, and the yield also ends when the replacement's last seen lease runs out on the old server's own clock. Reads are a single flight on the dedicated connection. While it yields, the server rechecks every 3 s, calls `onResume` when the marker is gone, and sends a coalesced queued-run-dispatch hint for skipped starts. - It yields only to a newer epoch with a live marker. It never yields to an older boot, to itself, or to a row without the marker, and it does not yield without a row of its own. - **Database invariant:** every statement of the hand-over runs in a small transaction on the dedicated one-connection controller-lease pool (`controllerLeaseDb`) with a 1 s statement timeout. That covers reads, marker writes, the `run-start-handover` notification (commit-bound, in the marker write's own transaction) and the skipped-start hint. None of it touches the shared pool or registers tracked database work. The pool is shared only with legacy controller-lease renewals, and never with the liveness pool that native lease renewals use. A hand-over statement waits at most one legacy batch (10 s), inside the 15 s between a marker renewal and the marker's expiry. It delays a legacy renewal (60 s lease, renewed every 10 s) by at most 1 s. I did not piggyback on the controller-instance lease renewal, because that renewal runs only while the server holds work and only every 30 s. - **Import audit of `run-start-handover.ts`:** - `drizzle-orm` `sql`: builds statements, no I/O. - `controllerLeaseDb`: the dedicated connection, used by every statement here. - `notifyDatabaseWorkOnCommit` (new in `work-signals.ts`): runs `pg_notify` only on the transaction it is given and registers no intent, so the coordinator's `reconcileDatabaseWork` has nothing of ours to check. - `logger`: no I/O to the database. - `controller-process-identity`: in-memory boot id and host hash. - `delivery-work-coordinator`: type only. The worker is registered for lifecycle and deadlines; its `run` awaits only the serialized marker chain, and reads are not awaited. - `DELIVERY_QUEUES`: constants. - `task-admission` drain subscriptions: in-memory events. - `heartbeat.ts`: run admission is also closed while the server yields. A `runStartHandover` service option (default: the process's registered hand-over) lets one process host two assembled services in tests. `queue.ts`: `startNextQueuedRunForAgent` and `resumeQueuedRunsAfterDrain` first call `refreshRunAdmission()`. That call returns `null` (nothing to await) unless the hand-over read is stale. - `index.ts`: creates the hand-over only when the instance lease is on, the deployment notification channel exists, and the scheduler is enabled. It calls `begin()` after the RunDispatch startup pass and wakes that queue. It releases the hand-over when run admission closes at shutdown, and stops it before the lease stops. - `deployment-notifications.ts`: keeps the LISTEN connection open while a hand-over involves this server, so skipped starts and lease changes reach the other server at once. - `doc/execution-semantics.md`: new section "Run starts during a deploy overlap". It covers the marker, who yields, failure handling, what counts as starting a new run, and what a post-deploy audit should show. **Marker contract for other components (for example the deployer):** a replacement starts runs when its `controller_instances` row has an `epoch` greater than the old boot's, and `dispatching_at is not null and dispatch_lease_expires_at > now() and stopped_at is null and fenced_at is null`. `dispatching_at` stays set after the lease lapses, as the time the boot began starting runs. ## Verification - `cd server && npx vitest run src/__tests__/heartbeat-run-start-handover.test.ts`: assembled-app test with two `heartbeatService`s, each with its own pool, listener, coordinator and boot. A wake that the old server accepts stays queued there. The replacement's real queued-run dispatch starts it, and claims are observed per pool. After a replacement release, the old server starts the run itself. Red: with the yield removed from `isRunAdmissionClosed`, the old server claims the run. - `cd server && npx vitest run src/__tests__/run-start-handover.test.ts`: 19 tests pass against embedded PostgreSQL. Each test uses two servers, each with its own pool, boot id, LISTEN connection and delivery coordinator. The tests cover: - the old server is ready and the new server starts; the new server sets the marker; the old server yields; the queued run is claimed only by the new server, and a skipped start wakes the new server's dispatch - the old server learns from its own dispatch path without a notification: a read older than 2 s is refreshed before the start - the new server dies after it sets the marker: the old server resumes and runs its queued-run pass - the new server shuts down: the old server resumes at once - the new server drains: it releases the marker and takes it back afterwards - a replacement that starts up during a drain takes over when the drain ends - a shutdown release during an in-flight renewal stays released - a clear that fails at shutdown (statement timeout behind a row lock) is retried by a later release - an older server that outlives the lease TTL keeps yielding while the replacement renews - a yield that the old server cannot confirm within 1 s (its only connection is busy) does not stop starts (red/green) - a yield ends when the replacement's lease runs out, even while every read waits for a connection (red/green) - renewals blocked past the TTL hand starts back once, then the replacement takes them again (bounded flapping) - a clear that fails when the replacement starts draining is retried well inside the lease (red/green) - with both servers' shared pools blocked past the TTL, the marker is written, renewed and released, and the old server yields and resumes, all before the block ends (red when hand-over statements use the shared pool) - an older-release row without the marker changes nothing - a server never yields to an older boot or to itself, and does not yield without a row of its own - a lone server writes once, never renews, and admits from memory - without a hand-over (self-hosted) there is nothing to await - Red: on `origin/master` the test file cannot run, because the module and the columns do not exist. The old server has no signal and keeps starting the handed-over run. - `cd server && npx vitest run src/__tests__/deployment-notifications.test.ts src/services/heartbeat/dispatch-work.test.ts src/__tests__/delivery-work-coordinator.test.ts src/__tests__/controller-instance-lease.test.ts src/shutdown.test.ts`: all pass. - `cd packages/db && pnpm check:migrations`: safety and compatibility checks pass. `cd server && npx tsc --noEmit -p .`: clean. - `pnpm verify` asks for broad verification because `doc/execution-semantics.md` is a global CI input. Required CI supplies that coverage. ## Risks - Run starts move from the old server to the new server during the overlap. If the new server cannot dispatch, the old server resumes on the lease clear (about 3 s) or the lease lapse (up to about 23 s after a crash). If a hand-over read fails, the server keeps admitting. - If the old server was not listening for notifications, it can start runs for up to about 2 s after the marker (the age of its last read). On a dispatching server, the pre-dispatch read adds one small bounded query at most every 2 s. - An older boot that has been idle for more than 30 min counts as gone. The new server then does not renew, and the old server admits again as today. - The migration adds nullable columns only. The deployer can apply it while the previous release serves. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits) and extended thinking. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
2174f2dfa5 |
perf(server): stop re-reading and re-decrypting unchanged checkpoint headers on every upgrade readiness probe (#21130)
Cloud repeats the upgrade readiness probe. Every probe loaded, decrypted and parsed the checkpoint header of each active native run and of each retained task session's last run, even when nothing had changed. The statement that reads a run now also reads its header's row revision (xmin, ctid, updated_at, stored size) and returns the material only for a revision the process has not inspected. A bounded memo in the checkpoint owner keeps the four identifiers the answer compares. It reuses an answer only for the same binding, row revision and secrets key, loads the key again for each answer it serves, never remembers a header that failed to decrypt, and forgets a header the latest probe did not inspect. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
f2e449f8f6 |
fix(retry): continue an accepted Retry past chat input held because it predates the stop (#21129)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A person's Retry on a connected task (for example a GitHub pull request review) must start work. #21080 made a Retry whose chat replay is refused after the click continue from the recovered saved work > - After #21080 was deployed, accepted Retries on a busy GitHub review agent still did not start > - Those tasks had a saved chat request that arrived while the failed run was still running. Admission holds such a request as `message_predates_stop`, so it can never start alone > - #21080 refused to continue while any saved chat input existed, so this held input blocked the continuation. For a second group, the latest run was an admission that the recovery hold cancelled before dispatch, and the recovered saved work belongs to the earlier failed run > - This pull request lets the continuation adopt input that was held because it predates the stop, and lets the recovered run that held a cancelled admission count as the saved work > - The benefit is that an accepted Retry starts work on these tasks too ## Linked Issues or Issue Description Follow-up to #21080. No public issue exists. This is the bug description, in the fields of `.github/ISSUE_TEMPLATE/bug_report.yml`. **What happened?** After #21080 was deployed, `POST /api/issues/:id/retry` returned 202 for blocked tasks of a GitHub review agent (`paperclip_runner`), but no run started. Each Retry receipt changed to `failed` about 5 seconds later (on the 14:48 Retries, 5 seconds for one task and about 75 seconds for two others), with no run and no `issue.retry_continued_from_saved_work` activity. Every affected task had a saved chat request (a newer GitHub review request) with an `issue.execution_waiting` record of reason `message_predates_stop`. The request arrived while the failed run was running, so admission holds it permanently. The interruption projection kept pointing at that held request. In a second group of tasks, the latest run was cancelled before dispatch with `execution_reconciliation_required`, and the recovered saved work belonged to the earlier failed run that held it. Root cause: in `continueRefusedTaskChatRetry` (`server/src/services/task-retry.ts`), any saved chat input from `selectTaskRetryInput` stopped the continuation, so the receipt settled `failed`. The held input can never start by itself, so nothing ran. The recovered-work check also looked only at the latest run, so it missed the recovered run that held a cancelled admission. **Expected behavior** An accepted Retry starts work. Saved input that admission holds because it predates the stop joins the continuation, and the task continues from the recovered saved work. **Steps to reproduce** 1. Bind a task to a GitHub chat conversation. Let a native run fail with `native_continuation_requires_reconciliation`, and recover its saved work. 2. Save a chat request while that run is still running, so admission holds it with `message_predates_stop`. 3. Click Retry while the chat identity lock is busy, so the source check is deferred and then refused. 4. The route returns 202. The Retry receipt changes to `failed`, and no run starts. **Paperclip version or commit** |
||
|
|
d5bd38468d |
fix(runner): stop a leftover Runner on the same state folder before a relaunch (#21079)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Agent turns run in a remote Runner (`paperclip-runnerd`) in a
Daytona sandbox. The Runner keeps its saved state in one folder per
session and listens on one port in the sandbox.
> - During an automatic upgrade, the old server can launch a Runner a
few seconds before it stops. That Runner stays alive in the sandbox and
keeps the state folder and the port.
> - The new server then restarts the run on the same sandbox. Each new
Runner fails at once with "Address already in use"
(`runner_ingress_bind_conflict`). Each launch also overwrites the
process marker that pointed at the old Runner, so the next check cannot
find the old Runner to stop it. The run restarted about 12 times in 19
minutes and then stopped, and the task stayed blocked.
> - Two Runner processes must never use the same saved state at the same
time.
> - This pull request makes each Runner launch stop any live Runner that
still names the same state folder, before the new Runner starts.
> - The benefit is that a restart after an upgrade gets the port and
continues the run, instead of looping on the bind conflict.
## Linked Issues or Issue Description
**What happened?**
On staging, a Runner launched by the old server just before an upgrade
kept the Runner port. Every relaunch by the new server exited within
seconds with "runner_ingress_bind_conflict: failed to bind listener:
Address already in use (os error 98)". The run looped for about 19
minutes and then failed with `native_runner_process_exited`. The task
was blocked and needed a person.
**Expected behavior**
The relaunch stops the leftover Runner of the same session, gets the
port, and continues the run.
**Steps to reproduce**
1. Launch a Runner for a run in a sandbox.
2. Stop the server before the Runner is adopted (an upgrade). The Runner
stays alive.
3. Let the new server restart the run's bootstrap. The new Runner cannot
bind the port, and the next restart cannot see the old Runner because
the marker now names the dead new process.
## What Changed
- `native-session-executor.ts`: the detached Runner launch script now
runs a short fence before it writes the process marker.
- It finds processes whose argv[0] is `paperclip-runnerd`, that are not
a `reap-lineage` command, and whose `--state-dir` is followed by this
exact folder. Runners on another folder, a folder named by a different
flag, and other programs are not touched.
- It records the stale Runner's lineage by PID **and start time**: its
descendants, and members of a process group or session that a recorded
process leads while that leader is alive (provider tools start their own
sessions). It signals a PID only while its start time still matches, so
a reused PID is never signalled, and it never signals a whole process
group.
- TERM, up to 5 s wait, then KILL, up to 2 s wait. Then it runs
`<runnerd> reap-lineage --state-dir … --runner-pid …` for the stopped
Runner, which stops the tools its lineage journal recorded that have
already left the process tree.
- It fails closed: a survivor (`runner_stale_state_holder_alive`) or a
reap that leaves a recorded process alive or gives no report
(`runner_stale_lineage_unproven`) makes the launch exit 75 **before**
the marker is written, so the existing marker is kept. The reason goes
to the process stderr log only when that path is a plain file (same rule
as the Runner stderr log). The whole stop is bounded at about 14 s,
under the 20 s marker wait.
- Two reap refusals let the launch continue with a
`runner_stale_lineage_partial` warning: `journal_unavailable` (the
Runner kept no journal; its visible lineage is already gone) and
`unattributed_orphan` (reported only after every recorded process is
gone, for orphans the reaper can neither claim nor signal) **only when
every orphan it names is proven to have another owner**: this launch's
own detached shell, a `paperclip-runnerd` serving a different
`--state-dir`, or that Runner's stderr trimmer. Any other orphan could
be an old tool that escaped between samples, so the launch fails closed.
- When `/proc` is not readable, or `grep` is missing, the fence does
nothing (the old behavior).
- `native-session-executor.test.ts`: Linux tests (a stale Runner whose
tool runs in its own session, plus four look-alikes that must survive;
fail-closed with the marker kept, with a plain and a symlinked log; reap
refusals for orphans owned elsewhere, an unknown orphan, and no journal;
a normal launch starts no extra node process) and a cross-platform check
that the embedded script parses and has no shell-special characters.
**When a launch happens vs. a reattach.** The launch is reached only
after the server chose not to adopt a Runner. A same-run reattachment
(`reattach_remote_runner`) first verifies the marker's Runner: if it is
alive it is adopted and nothing launches; a transient verification
failure (for example the read timeout that #21053 keeps the Runner alive
for) throws and never reaches a launch; only a Runner proven not running
falls through to a fresh launch. Warm recovery likewise throws instead
of launching when revalidation fails. Every launch overwrites the
marker, so after a launch no reattachment could adopt a leftover Runner
anyway, and its port would make the new Runner fail with
`runner_ingress_bind_conflict`. Stopping it therefore never removes a
Runner that the same run could still re-adopt.
## Verification
- Linux (Debian bookworm container, node 24): the Linux tests pass on
this head. The same tests against the previous head (
|
||
|
|
4e932fd178 |
perf(attention): index the agents and review paths once for every review of a request (#21132)
<!-- Write all pull request text in Simplified Technical English (ASD-STE100): short sentences, one instruction per sentence, simple approved vocabulary, and the active voice. --> ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A task in review needs a path to its next action: a reviewer, a pending interaction or approval, a monitor, a run, a wake, an external reply or a recovery. A review without a path is stalled, and the attention feed tells a person about it. > - The server decides this for each task in review (`listIssueReviewAttentionMap` in `server/src/services/issues.ts`, with the classification in `server/src/services/recovery/issue-graph-liveness.ts`). The task list, the blocked inbox and the attention feed call it. `GET /attention/summary`, which the board sidebar reads on most pages, runs it for every task in review of the company. > - The classification did the company-wide work again for each task. Each check of an agent copied the agent list, indexed it again and walked the reporting chain. Each task scanned every run, wake, interaction, approval, external wait and recovery of the request. Each stalled review also filtered and sorted every agent again, with one such check for each agent. > - Thus the work grew with reviews × agents for covered reviews and with stalled reviews × agents² for stalled reviews. It also allocated a new list and a new index for each check, and the garbage collector of the server paid for that. > - On a hosted instance, an allocation sample of the server put 4.7% of all allocated bytes under this function, and garbage collection used 25% of the main thread. The two slowest board requests there are the attention summary and the task list, which both call it. > - This pull request builds one index for one classification: the agents by id, the eligibility of each agent, the invokable agents of each company in id order, and the paths of each task. Each task then costs its own paths. > - The benefit is that the review classification of a summary request falls from 62 ms to 5 ms for 900 reviews and 60 agents, and from 123 ms to 8 ms for 300 reviews and 180 agents. The request allocates 57% to 91% fewer bytes. Every finding, every review path and every count is the same. ## Linked Issues or Issue Description Refs #20851 and #20858 (they batched the review-wait provenance check and the task read decision of the same requests; this change is the next cost on that path). Refs #19194 and #13729 (open; they change the failed-run lookup of the attention feed, which is a different statement and a different mechanism; this pull request does not touch it). **What existing behavior does this improve?** The time and the memory that the server uses to decide the review attention of the tasks in review, for `GET /api/companies/:companyId/attention/summary`, `GET /api/companies/:companyId/attention`, `GET /api/companies/:companyId/issues` and the blocked inbox. **Subsystem affected** Server: task-graph liveness classification (`server/src/services/recovery/issue-graph-liveness.ts`) and the review-attention map in `server/src/services/issues.ts`. **Current behavior** - `classifyIssueReviewPaths(input, issue)` built `new Map(input.agents…)` for each task and scanned six path lists of the request for each task. - `isInvokableAgent` copied the agent list for each check. `getAgentOrgChainHealth` then indexed that copy again. - `ownerCandidatesForRecoveryIssue` ran for each stalled review. It filtered and sorted all agents, and it checked each agent again in two more loops. Each check did the copy and the index above. - `hasActiveExecutionPath` built a joined list of the runs and the wakes for each blocker it looked at. - `listIssueReviewAttentionMap` searched the review list for each open recovery task (`reviewIds.includes`) and for each review (`issues.find`). Evidence from a hosted instance (sanitized): - Server main thread at 93% of one core. Garbage collection: 25% of the main thread (387 pauses in 32 s, the longest 224 ms). - Allocation sample, 30 s: 77.7 MB/s in total. `listIssueReviewAttentionMap` and its callees: 4.7% of all bytes (`classifyIssueReviewPaths` 4.4%, `classifyIssueGraphLiveness` 2.3%, `reviewFinding` 2.3%). - `GET /attention/summary` took 16.3 s of server time and `GET /issues` up to 15.3 s. Most of that time is waiting behind other work on the same thread (see "Not verified"). **Proposed behavior** One classification indexes its input one time. The cost of one task is the number of its own paths. The cost of the agents is one eligibility decision for each agent. The results do not change. **Reason and benefit** The summary request runs on most board pages. Its CPU time and its allocation also delay every other request, because the server has one main thread and one garbage collector. **Breaking changes** None. `classifyIssueGraphLiveness` and `classifyIssueReviewPaths` keep their signatures and results. `packages/shared` does not change. ## What Changed - `server/src/services/recovery/issue-graph-liveness.ts` - New internal index for one input (`createIssueGraphLivenessIndex`): agents by id, a memo of `getAgentWorkEligibility` for each agent record, the invokable agents of each company in id order, and the paths of each task for the six path lists. The path lists are indexed on first use and keep the order of the input. The company check of each path stays. - `classifyIssueGraphLiveness` builds the index one time and uses it for the review paths, the waiting paths of blockers, the owner candidates and the assignee eligibility of blockers. - New export `createIssueReviewPathClassifier(input)`. It returns the review paths of many tasks from one index. `classifyIssueReviewPaths(input, issue)` stays for single tasks. - The memo uses the agent record as its key, not the agent id. Thus an input that lists one id two times gets the same answers as before. - The eligibility decision itself is still `getAgentWorkEligibility` of `@paperclipai/shared`, with the same agent list. The index calls it one time for each agent, not one time for each check. - `server/src/services/issues.ts` (`listIssueReviewAttentionMap` only): one classifier for all reviews of the request, a set for the review ids, and the review by position instead of a search. No query changes. - Tests and the reproduction (see Verification): - `server/src/__tests__/helpers/issue-graph-liveness-reference.ts`: the classification as it was before this change. It is the oracle of the equivalence tests. - `server/src/__tests__/issue-graph-liveness-equivalence.test.ts`, `server/src/__tests__/review-attention-classification-equivalence.test.ts`, `server/src/__tests__/helpers/review-attention-fixture.ts`. - `server/scripts/benchmark-review-attention.ts`: the reproduction through the real routes. ## Verification Reproduction and before/after measurement. The script starts a local PostgreSQL, seeds a synthetic company, mounts the real `attentionRoutes` and `issueRoutes` with a board user, and sends the two requests over HTTP. With `--baseline-ref` it runs the pre-fix modules in a second process against the same database. It fails unless both versions return the same summary, the same full attention feed, the same task list and the same review attention for every task in review. ```sh pnpm --filter @paperclipai/server exec tsx scripts/benchmark-review-attention.ts --baseline-ref=ef2b53b675feebb1bfbd6701627f6c302f8c6916 --tasks=3000,9000 --agents=60 pnpm --filter @paperclipai/server exec tsx scripts/benchmark-review-attention.ts --baseline-ref=ef2b53b675feebb1bfbd6701627f6c302f8c6916 --tasks=3000 --agents=180 ``` Fixture: one company with tasks of every status (10% in review, 10% blocked), four of five tasks are children, a reporting tree of agents with paused, terminated and pending agents. The reviews cover every kind of path and three ways to stall (31% stalled). The blocked tasks wait behind chains, cycles, cancelled, done, backlog and unassigned blockers. Open recovery tasks (5% of the tasks) name reviews, blocked tasks and done tasks. 6% of the reviews wait for a GitHub reply. A second company with the same shapes is in the same database. Embedded PostgreSQL 18.1, Node v26.4.0, `VACUUM ANALYZE` after seeding. Each version ran in a new process, three rounds in alternating order, 2 warm-up requests and 5 measured requests in each round. A CPU profile of 3 more requests gives the last two columns. | Tasks / agents / reviews | Request | Version | Statements | Server ms, median (min) | CPU ms, median (min) | Main thread busy ms, median (min) | Allocated MB | Review classification ms | GC ms | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | 3000 / 60 / 300 (94 stalled) | attention/summary | before | 69 | 121.4 (112.7) | 131.5 (114.4) | 80.2 (73.2) | 170.4 | 16.8 | 5.0 | | 3000 / 60 / 300 (94 stalled) | attention/summary | after | 69 | 105.9 (94.7) | 113.3 (96.3) | 67.3 (58.6) | 73.3 | 2.4 | 3.4 | | 9000 / 60 / 900 (283 stalled) | attention/summary | before | 85 | 280.9 (263.9) | 281.4 (253.4) | 203.7 (186.8) | 509.8 | 62.1 | 30.0 | | 9000 / 60 / 900 (283 stalled) | attention/summary | after | 85 | 224.8 (213.4) | 220.1 (205.8) | 149.9 (142.3) | 219.1 | 4.8 | 6.6 | | 3000 / 180 / 300 (94 stalled) | attention/summary | before | 69 | 241.2 (203.9) | 248.7 (227.4) | 197.1 (171.1) | 950.2 | 123.2 | 21.2 | | 3000 / 180 / 300 (94 stalled) | attention/summary | after | 69 | 127.8 (108.5) | 128.5 (102.0) | 76.7 (64.8) | 87.3 | 7.8 | 5.1 | | 3000 / 60 / 300 | task list (500 rows, 48 reviews) | before | 59 | 102.7 (96.7) | 92.4 (83.6) | 54.7 (48.9) | 73.2 | 2.6 | 2.4 | | 3000 / 60 / 300 | task list (500 rows, 48 reviews) | after | 59 | 106.2 (91.2) | 89.9 (78.8) | 55.2 (44.8) | 63.4 | 0.9 | 1.6 | | 9000 / 60 / 900 | task list (500 rows, 50 reviews) | before | 42 | 194.3 (186.0) | 77.5 (69.1) | 43.3 (38.4) | 57.3 | 1.0 | 1.8 | | 9000 / 60 / 900 | task list (500 rows, 50 reviews) | after | 42 | 189.6 (182.3) | 72.1 (69.5) | 41.9 (39.4) | 57.8 | 1.2 | 2.5 | | 3000 / 180 / 300 | task list (500 rows, 48 reviews) | before | 59 | 124.6 (95.7) | 110.2 (99.4) | 67.8 (58.1) | 165.4 | 15.4 | 3.8 | | 3000 / 180 / 300 | task list (500 rows, 48 reviews) | after | 59 | 104.1 (91.9) | 90.7 (79.9) | 56.2 (47.1) | 78.2 | 2.9 | 5.3 | How to read the table: - "Review classification ms" is the time in `listIssueReviewAttentionMap` and everything it calls on the main thread, for one request, from the CPU profile. It is the direct measure of this change. Before the fix it was the largest block of application code in the summary handler: 22% of the busy time at 3000 tasks, 20% at 9000 tasks, and 61% with 180 agents. `getAgentOrgChainHealth` under `isInvokableAgent` under `ownerCandidatesForRecoveryIssue` had most of it. After the fix it is 4% to 9%. - The scaling: three times the tasks gave 3.7 times the classification time before and 2.0 times after. Three times the agents gave 7.3 times before and 3.3 times after. - "Allocated MB" is all bytes that the server process allocated for one request (`total_allocated_bytes`), including the HTTP client of the script. - Other agents used the same machine during the measurement (load average 8 to 16 on 18 CPUs), so the medians of server time have noise. The profile columns and the allocation column are the stable ones. - The task list changes little with 60 agents, because one page has about 50 reviews. With 180 agents its classification falls from 15 ms to 3 ms and its allocation by half. - Statement counts do not change: this change removes no statement. Red/green regression test for the mechanism (`server/src/__tests__/issue-graph-liveness-equivalence.test.ts`): - `reads each agent and each path a number of times that does not grow with the reviews`. The test counts the reads of the agent status and of the path task id for 20 and for 400 stalled reviews with 40 agents and 360 paths. On the code before this change: `expected { agentStatus: 200800, pathIssueId: 144000 } to deeply equal { agentStatus: 10040, pathIssueId: 7200 }`. After: 316 agent reads and 360 path reads for both sizes (one read of each path). The test also runs the oracle and asserts that its reads grow, so it shows what it catches. - `classifies the review paths of a page from one index`. The same count for `createIssueReviewPathClassifier`. Equivalence tests (old code and new code on the same input): - `returns the findings and review paths of the pre-fix classification for random task graphs`. 400 seeded random graphs of two companies. Each graph has reviews with and without paths, blocked chains and cycles, terminal blockers, relations and paths that name the other company or a missing task, reporting chains with cycles, missing managers and managers of the other company, and ids that the input lists two times. The test compares the findings of the graph and the review paths of every task, for both `classifyIssueReviewPaths` and the new classifier. It asserts that the graphs reach all six finding states and all nine kinds of path, each at least 20 times. - `returns the pre-fix findings for blocked chains, cycles and terminal blockers`. One named graph with the expected state of each finding, so the comparison cannot pass on two wrong answers. - `server/src/__tests__/review-attention-classification-equivalence.test.ts`, on real PostgreSQL through the real routes. The test wraps the classification module: every classification that a request runs is also run by the oracle on the same input, and a difference fails the test. - `gives every review the paths the fixture was built with, and the pre-fix classification of every request`. `GET /issues?status=in_review`, `GET /issues` and `GET /issues?attention=blocked` (the blocked inbox, whose graph has the chains, cycles and terminal blockers). Each of the 64 reviews has the path kinds and the stalled reason that the fixture states for it, including reviews that wait for an external reply and reviews that open recovery tasks cover. - `counts the reviews that need a person, leaves out the ones a person dismissed, and reads one company`. `GET /attention/summary` and `GET /attention`: the review count and the review cards are the reviews that the fixture built to need a person. The second company has its own counts and none of the first company's tasks. Three dismissed cards leave the count and stay stalled in the task list. - The benchmark above compares the complete responses of both versions through the routes at each size (summary, full feed, task list, review attention of every review). Commands run locally: - `pnpm --filter @paperclipai/server exec vitest run` for `issue-graph-liveness-equivalence`, `review-attention-classification-equivalence`, `issue-liveness`, `recovery-classifiers`, `issue-review-attention`, `issue-blocker-attention`, `blocked-inbox-review-wait-scope`, `heartbeat-issue-liveness-escalation`, `attention-retention-work` and `attention-service`: 210 tests passed. In that run the `afterAll` hook of `attention-retention-work` and of `attention-service` (the stop of the test database) timed out at 30 s while other agents loaded the machine. No assertion failed. CI runs both files. - `tsc --noEmit` for the server package: no errors. The new test files and the script are outside the server `tsconfig`; a focused `tsc` program for them reports no error in these files. - `pnpm check:module-boundaries`, `node scripts/check-no-git-push.mjs` and `node --test scripts/__tests__/run-vitest-stable-shard.test.mjs`: pass. - The first CI run of this pull request failed `ci / Runner support suite`. An earlier revision also let the eligibility functions of `packages/shared` accept a prebuilt index, and the source of `packages/shared` is pinned for the runner SDK source proof (`tests/runner-e2e/source-sdk-public-inputs.json`). The fix does not need that change, so the current revision leaves `packages/shared` as it is on `master`: no pin moves, and the UI, the CLI and the runner use the same eligibility code as before. `source-sdk-public-inputs.test.mjs`, `source-sdk-evidence.test.ts` and `source-sdk-public-type.test.ts` pass locally on this revision (58 tests), and the drift guard reports no stale pin. - `pnpm verify` returns exit 2 (broader verification required) because the change adds a file under `server/scripts/`. CI supplies the broad coverage. The repo-wide typecheck, build and test suite were not run locally (shared machine). Not verified: - The effect on the hosted instance. This pull request does not deploy. The local requests take 0.1 s to 0.4 s, and the hosted requests took 15 s. Thus most of the hosted time is not this code: the hosted main thread is saturated by other work, and each of the 69 to 85 statements of a summary request waits for its turn on that thread. This change removes CPU time and allocation from that thread. It does not remove a statement. - The number of agents and of stalled reviews of the hosted company. The hosted allocation sample has most of the bytes in `classifyIssueReviewPaths`, which is the cost for each review (reviews × agents). The local fixture has more stalled reviews, where the owner candidates cost more. - A remaining cost inside the classification: each eligibility decision of `@paperclipai/shared` indexes the agent list again. One classification now does this one time for each agent (agents² map entries for one request, 3 ms with 180 agents), not one time for each check. The helper can accept a prebuilt index in a later change; that change moves the SDK source pins, so it needs its own review. - A second, independent cost that this pull request does not change: `GET /attention/summary` returns counts only, but it builds the full feed. After this change, the largest application blocks in its profile at 9000 tasks are the full task list of the 900 blocked tasks (15 ms on the main thread, of which 9 ms decode description previews that the summary never returns), the blocker attention of those tasks (9 ms), and the text of its 85 statements. A later change can give the summary a count-only path. ## Risks - The attention feed decides what a person is told needs them. The risk is a classification that differs from the previous one. Mitigation: the rules did not move. Only the lookups changed (a map lookup instead of a scan, a stored eligibility instead of a new one). The oracle tests compare the findings and the paths for random graphs, for a named graph and for every classification of the route tests. The benchmark compares the full responses. - The index is a snapshot of the input. `createIssueReviewPathClassifier` documents that the input must not change while the classifier is in use. The two callers build the input and use it in the same synchronous block. - When the input has no `now`, one classification now reads the clock one time. Before, `classifyIssueGraphLiveness` read it one time for the blocker checks and one more time for each review. All callers in the server that classify many tasks pass `now` or run in one synchronous call, so the values differ by less than the run time of the call. - The oracle file is a copy of the previous classification (about 640 lines, test-only). A later change of a classification rule must change the oracle in the same way. The header of the file says so. - The repository rule in `CONTRIBUTING.md` says that a pull request has no code comments. This pull request has comments where the code has a constraint that the names do not show (the memo key, the snapshot rule, the purpose of the oracle, the usage of the script), as the merged changes around this code do. - No migration. No query change. No change to company scope or to actor permissions: the change is after the reads, on rows that the same queries return. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude Code with tool use (code reading, editing, shell, tests, CPU profiles). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>canary/v2026.1112.0-canary.13 |
||
|
|
df89950eb6 |
fix(native): save or set aside a superseded run's unsaved workspace so its task queue moves (#21136)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native run saves its accepted result. Then a finalizer exports the run's workspace files and applies the result to the task. > - When a newer run already decided the task, the finalizer retires the older result as superseded. Since #21106, this also ends the run. > - If that retirement interrupts a failed workspace export, the run's files stay unsaved. The run also stays the writer of the task's workspace. Nothing records those files for recovery, so nothing ever releases the workspace. > - Every run queued on the task then waits behind a save that never settles. After one hour, each queued run fails setup instead. > - This pull request records those files for the existing automatic workspace recovery. Recovery saves them from the retained sandbox, or, when every bounded attempt fails, continues the task from its last save with the files attached. > - The benefit is that the task's queue moves again by itself, and the run's unsaved work is saved or kept findable. The superseded result is never applied. ## Linked Issues or Issue Description Refs #21106 **What happened?** On a hosted staging server, a native run accepted its result at 11:31:08Z. Its workspace export failed once with `remote_workspace_archive_fs_eacces`. The finalizer then retired it as `native_finalization_superseded`, because another run had already decided the task. #21106 ended that run at 14:59:17Z (`failed`). The task still could not move. The next queued run was cancelled with `workspace_busy` ("ended, but its workspace save has not settled"). Its retry re-armed every minute. Four more runs stayed queued. The task showed no recovery action. **Expected behavior** The ended run's unsaved files are saved from its retained sandbox, and the workspace is released. If they cannot be saved within the bounded retries, the task continues from its last committed save, and the files are attached to the task. The queued runs proceed in both cases. The superseded result is never applied. **Steps to reproduce** 1. Accept a native result for a run that writes a remote workspace. 2. Let the workspace export fail once, while a newer run has already committed the task's decision. 3. See that the finalizer is `terminal_failure` / `native_finalization_superseded` and the run is `failed`. 4. See that the workspace still names the run as its dirty writer, and that no retained-file recovery action exists. 5. Queue another run on the task. It defers with `workspace_busy`, and after one hour it fails setup with "The workspace still has an unfinished writer or save operation." **Root cause** `recordRetryableFailure` (`native-run-finalizer.ts`) retires a superseded finalizer without preserving the export. `preserveNativeFailedWorkspace` records a retained-file hold for an accepted result only when the failure code is `native_workspace_sync_out_retry_exhausted`. A `native_finalization_superseded` coordinator never qualifies. `readRemoteWorkspaceWriterHold` keeps an ended dirty writer as the holder until its save settles. Without a hold, no worker ever settles it. **Paperclip version or commit** Staging Core with #21106 (`81d2e4d8d9`). Master `ef2b53b675` has the same behavior. ## What Changed - `native-superseded-workspace-export.ts` (new): identifies a superseded finalizer that failed in the workspace export. It selects ended runs that still hold their workspace with unsaved files and no retained-file hold. It records the hold through `preserveNativeFailedWorkspace`. It checks again only while a just-ended run's sandbox stop receipt is still arriving (at most one hour). - `native-failed-workspace-recovery.ts`: `preserveNativeFailedWorkspace` treats a superseded workspace export like an exhausted copy-back, with the same proofs (run ended, no live controller, accepted result, confirmed sandbox stop-and-retain, source still bound to the workspace). It marks the hold `supersededWorkspaceExport`. - `native-run-projection-work.ts`: the native run projection worker also selects these runs. It loads them at startup and wakes when a finalizer changes. Rows written by older releases are therefore settled after deploy without a person. - `workspace-recovery-fallback-discovery.ts` and `workspace-recovery-fallback.ts`: a marked hold whose bounded re-export is spent (at most 5 retries in 30 minutes) is eligible for the existing continue-from-last-save fallback. That fallback sets aside only this run's workspace epoch and keeps its location on the workspace record. It attaches the readable files to the task unapplied. The task message says Paperclip "could not save" the previous run's unsaved files. - `preserved-result-finalization.ts`: a superseded result is never resumed for application, even after its files are saved. - `doc/execution-semantics.md`: document the rule. - No schema change. Nothing changes on the path that starts a run or a turn. ## Verification - `server/src/services/superseded-workspace-export-recovery.test.ts` (new). The first two tests fail on master and pass on this branch: - "records the files for automatic recovery, saves them from the retained sandbox and frees the workspace". It uses the staging row shape. The projection worker records the hold, and a second pass changes nothing. The workspace waits for that save. The system recovery saves `saved.txt` into a committed snapshot. The workspace becomes free. The finalizer, the result and the task's decision do not change, and the result is not resumed. On master the worker selects nothing (`expected 0 to be 1`). - "continues from the last save with the files attached when every re-export fails". After an exhausted re-export budget, the hold is resolved `restored`. The files are attached (`quarantine.state = captured`). The workspace moves to epoch 2 with no writer, and the abandoned epoch keeps its lease location. The message has no request for a person to fix anything. - "leaves a superseded result alone when it never failed in the workspace export". - Related suites pass: `native-run-projection-work`, `native-finalization-recovery`, `workspace-recovery-fallback`, `native-failed-workspace-recovery`, `workspace-recovery-retry-copyback`. - `tsc --noEmit` for `server`. ## Risks - Low to medium risk. The change acts only on runs whose finalizer is already terminal and superseded, that failed in the workspace export, and that still own a dirty workspace. All existing preservation proofs still apply. - The saved files of a superseded run become the workspace's next saved version. The run's result (the task decision) is still never applied. - When the export can never succeed, the task continues from its last save. Files that cannot be read are not attached. The workspace record keeps the set-aside epoch's location. Saving the readable part of a partly unreadable workspace is not in scope. - Runs that already failed setup before this deploys are not restarted by this change. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (code search, shell, test execution, read-only API inspection) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
782d6bead4 |
Configure managed plugin Runner lifecycle (#21065)
## Thinking Path > - Paperclip runs agents in managed environments. > - An agent can use an explicit environment or the instance default. > - A lifecycle plugin must receive the driver that provisions the agent. > - A managed plugin environment also needs to select the Runner process lifecycle. > - This change gives lifecycle selection and provisioning the same environment lookup. ## Linked Issues or Issue Description **What existing behavior does this improve?** Managed plugin environment provisioning and lifecycle configuration. **Current behavior** Lifecycle calls lack the selected plugin driver. Managed plugin configuration cannot select warm Runner execution. **Proposed behavior** Resolve the environment once per lifecycle operation. Pass the driver key for the plugin that serves it. Parse and persist the managed Runner lifecycle mode. The portable ownership and workspace placement changes are in [#21066](https://github.com/paperclipai/paperclip/pull/21066) and [#20743](https://github.com/paperclipai/paperclip/pull/20743). ## What Changed - Share agent environment resolution between required plugin selection and the lifecycle RPC. - Pass `environmentDriverKey` to the plugin that serves the selected environment. - Allow managed plugin environments to set `warm` or `per_turn` and persist the selected mode. - Advertise `pluginEnvironmentRunnerLifecycleMode` in the release capability manifest so senders can check support before including the field. ## Verification - Built shared types, the plugin SDK, and Runner. Server typecheck and SDK source-pin check passed for the initial layer. - The capability contract test failed before the manifest fix and passed after it. Independent verification passed all 160 tests across managed configuration, plugin environment persistence, lifecycle routing, and environment reconciliation. - Routing coverage includes the instance default, an explicit second driver from the same plugin, and an explicit sandbox. - [Hosted CI](https://github.com/paperclipai/paperclip/actions/runs/43193311862) passed on `f07b192c21`. The first attempt timed out in an unchanged workspace-recovery fixture and then collided with a background recovery action. The exact test and all 44 tests in its suite passed locally; the isolated shard retry passed without code changes. ## Risks - Existing managed configuration can omit the lifecycle mode. Senders must check the target release for `pluginEnvironmentRunnerLifecycleMode` before sending the new field. Publishing this signal does not update external deployment tools. ## Model Used - OpenAI Codex GPT-6 with GPT-6.1 Sol agents for implementation and independent review. ## Checklist - [x] Thinking path and model are included. - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] The issue is described using the enhancement template. - [x] Local checks passed and new behavior has tests. - [x] All Paperclip CI gates are green. - [x] Greptile is 5/5 with no open findings. --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
1af987690d |
perf(storage): hash and encrypt workspace snapshots off the server thread (#21138)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - One server process serves every request and agent turn on one thread. The same process saves each remote workspace as one encrypted archive in object storage, and restores it before a turn > - `server/src/services/remote-workspace-snapshot-storage.ts` hashed, encrypted and decrypted every byte of those archives on that thread. `server/src/storage/s3-multipart.ts` computed the Content-MD5 of each uploaded part on that thread too > - Hashes and ciphers are synchronous in Node. A CPU profile of a busy hosted server showed this work on the main thread, which was at 93% of one core > - Each megabyte of a snapshot delays every request and turn that waits behind it > - This pull request moves those passes to a small, bounded set of worker threads. The checks, their order, their failures and the stored format do not change > - The benefit is that a 1 GiB save uses about 0.7 s of the server's thread instead of about 2.8 s, and an unrelated request during the save is answered in 1.3 ms (p99) instead of 16.7 ms ## Linked Issues or Issue Description No public issue exists. I searched open and closed pull requests for `workspace snapshot hash encrypt`, `event loop` and `archive io`. Related merged work: #19900 gave these stream passes 1 MiB buffers, #20554 removed a read-back from migration, #20722 took runner artifact hashes off the event loop, #20709 records long event-loop blocks. No open pull request covers this change. **What happened?** A 30 second CPU profile of the main thread of a hosted server (commit |
||
|
|
fb8b61bc3e |
fix(native): restart a bootstrap whose closing server staged files but never launched its Runner (#21134)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agent turns run in a remote Runner (`paperclip-runnerd`) in a Daytona sandbox. A release swap stops the old server and a new server recovers each run. > - When the old server stops during a run's bootstrap, the new server restarts that bootstrap on the same run (`bootstrap_incomplete`). The claim is granted only when the Postgres controller checkpoint proves that no provider turn was sent. > - The old server can stop after it staged launch files (Codex home, config) into the sandbox session folder, but before it launched the Runner. Its detach fence stops it there. > - The new server then finds no Runner state in the sandbox. A fresh bootstrap must create the session folder itself, and the folder already exists, so the run failed with `runner_harness_state_mismatch: resumed_unavailable`. The failure is an ambiguous session, so the task waits for a person. > - This pull request lets host preparation prove that no Runner was ever launched in that folder, move the folder aside, and restart the bootstrap. > - The benefit is that a release swap during this part of a bootstrap no longer blocks the task. No provider work can be lost or replayed, because no Runner ever ran. ## Linked Issues or Issue Description Refs #21120 (same release-swap class: the Runner was launched and connected before the old server stopped; that pull request stops it in host preparation. This pull request covers the case where no Runner was launched.) Refs #21079 (launch-time fence for a stale Runner on the same state folder.) **What happened?** On hosted staging, during a release swap on 2026-11-12, a Codex run on Daytona failed with `native_session_interrupted` and `runner_ingress_unavailable: runnerd process failed: runner_harness_state_mismatch: resumed_unavailable`. The old server was in `harness_state.launch_material` when its shutdown drain detached the run. It recorded no Runner process identity. The new server chose "Restarting an incomplete native runner bootstrap on the same heartbeat run". Its host preparation found the sandbox resumed and no Runner state. Then the claim of the session folder failed, because the old server had already created that folder. The run needed a person. **Expected behavior** The run restarts its bootstrap on the same sandbox and continues without a person. A provider turn is never replayed and two Runners never share the run. **Steps to reproduce** 1. Start a Codex run on a Daytona sandbox. 2. Stop the server after the bootstrap stages launch files into the sandbox session folder and before it launches the Runner. 3. Let a new server recover the run with a `bootstrap_incomplete` claim. The new server has no local controller root for the run. 4. Host preparation fails with `runner_harness_state_mismatch: resumed_unavailable`. **Paperclip version or commit** Staging release `704b3f4d63`. The code is the same on master at `ef2b53b675`. **Deployment mode** Hosted staging, remote sandbox execution on Daytona. **Agent adapter(s) involved** Paperclip Runner with Codex (`codex_app_server`). ## What Changed - `native-session-executor.ts`: under a same-run `bootstrap_incomplete` claim with no failover backup, when host preparation finds no Runner state, a new check asks the sandbox in one command whether the session folder exists and holds no Runner trace. A Runner trace is the live `runner/runner-state.json`, the `runner/runner-process.identity` launch marker (the launch script writes it before the Runner starts), an `authority-epochs` archive, or any other `runner-state.json` or launch marker anywhere under the folder. If the folder holds no Runner trace, no Runner was launched there, so host preparation records the restart proof (`absent_confirmed`). It then uses the existing path: move the folder aside (it is kept for inspection), claim a new folder, and bootstrap again. If the folder is missing, the existing fresh-claim path runs unchanged. If any Runner trace exists, the run stays fail-closed as before. An archived session is therefore never discarded. - `native-session-executor.test.ts`: new `interrupted_bootstrap_unlaunched` cases. Resumed and created sandboxes restart the bootstrap. A launch marker, an archived authority epoch, Runner state elsewhere in the folder, or a failover backup keeps the run fail-closed with `resumed_unavailable`. ## Verification - Red: with the source change reverted, `pnpm exec vitest run server/src/services/native-runtime/native-session-executor.test.ts -t "bootstraps only an untouched provider session" --maxWorkers=2` fails on both `interrupted_bootstrap_unlaunched` restart cases with `runner_harness_state_mismatch: resumed_unavailable`, which is the staging error. - Red: without the archived-state guard, the `_archived` and `_nested_state` cases fail (the folder is moved aside). With the guard they pass. - Green: the same command passes (64 tests). - `pnpm exec vitest run server/src/services/native-runtime/native-session-executor.test.ts --maxWorkers=2`: full file, 1422 passed before the guard commit. The focused 64 pass on the final revision. - `npx tsc --noEmit -p server/tsconfig.json`: passes. - `git merge-tree` against #21120's head (`0d18f18928`) merges without conflict. The new check runs only when no Runner state exists, so #21120's new call site (it runs only when a Runner state exists) gets the same results as before. ## Risks - Low. The new path runs only in host preparation, under a `bootstrap_incomplete` claim for the same run id, with no failover backup. That claim is granted only after the previous controller's lease has ended. With no Runner state or launch marker anywhere in the folder, no Runner and no provider session existed there. The old folder is moved aside, not deleted. - A controller that is still launching at the same moment would have to hold a lease that the claim already took over. The detach fence stops such a launch, and #21079 adds a launch-time fence for a leftover Runner on the same state folder. - No added wait before a normal turn starts: the check runs only on this failure path. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, browser for read-only run evidence, code execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |