PaperClipAI

0 Fork 0

mirror of https://github.com/paperclipai/paperclip.git synced 2026-11-12 06:10:44 +01:00

Author	SHA1	Message	Date
DottaandClaude Opus 5.5	1baac27888	feat(server): chat bootstrap reads for instant task opens (#20355 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - People open tasks from the task list to see the latest messages and to reply to an agent > - Before the task chat shows its first frame, it waits for about ten separate reads: the issue, recent comments, comment input facts, recent activity, recent runs, interactions, attachments, live runs and the active run > - On a busy instance each read waits for the database, and the slowest read decides when the chat appears. A cold open takes 1.5–4 s on staging > - This pull request adds one read that returns all of that data for up to ten tasks, in the same JSON shapes as the existing endpoints, plus a single-task form > - The benefit is that the task list can preload its top tasks, and a cold open needs one round trip instead of about ten. The UI change is in the stacked pull request ## Linked Issues or Issue Description Problem or motivation A task open waits for about ten reads. With the database under load, these reads queue, and a cold open takes seconds. The list cannot preload a task cheaply, because each task costs about ten requests. Proposed solution `GET /api/companies/:companyId/issues/chat-bootstrap?ids=` returns the first-frame data for up to 10 tasks. `GET /api/issues/:ref/chat-bootstrap` returns it for one task. Each field equals the matching single endpoint's JSON exactly. Alternatives considered - Prefetch the existing endpoints for each list row. For 30 rows this sends about 210 requests to a database that is already the bottleneck. - A new projection with different shapes. This would duplicate client merge logic, and freshness rules could drift. Roadmap alignment This is part of the task-loading performance plan in #20276. It follows #20277, #20284 and #20286. ## What Changed - `packages/shared`: - `ISSUE_CHAT_BOOTSTRAP_MAX_IDS` (10). - `issueChatBootstrapQuerySchema`: UUIDs or identifiers, normalized, deduplicated and strict. - The `IssueChatBootstrap` and `IssueChatBootstrapItem` types. - New `server/src/services/issue-chat-bootstrap.ts`: - Resolves the refs inside the company and applies the `issue:read` decision to each task. - Reads the tasks with a concurrency of 2, and returns them in request order. - `server/src/routes/issues.ts`: - The core issue read and the recent-comments read are now shared helpers, so the bootstrap and the single endpoints use the same code. The core view keeps its Server-Timing header. - Adds the batch route and the single-task route. - Excludes the single-task route from the issue ETag middleware, so a fetch without validators always gets a 200 with `Cache-Control: private, no-store`. - New `server/src/services/issue-run-presence.ts`: - The live-runs and active-run payload builders, moved from `agents.ts` without change. - The existing routes call them, and their behavior is unchanged. - `server/src/routes/openapi.ts`: entries for both routes. ## Verification - `cd server && npx vitest run src/__tests__/issue-chat-bootstrap-routes.test.ts` passes 17 tests. They run on embedded Postgres with privacy enforcement on. They cover: - Each field deep-equals its single endpoint's JSON, for an ordinary task and for a conversation task. - The single-task route equals the batch item, by UUID and by identifier. It is never answered with a 304. - Items keep the request order, with UUIDs and identifiers mixed. - Private, other-company, missing and unknown tasks are omitted, and none of their data appears. The single-task route returns 404 for them. - Malformed, extra or repeated parameters return 400. More than 10 ids also return 400. - A task-bridge key gets 403. - A 10-task request writes nothing: no inserts, updates or deletes, and no activity-log or comment changes. - The issue, agent, activity, live-run, cold-open, summaries, ETag and OpenAPI route suites pass: 56 files, 1,771 tests. - `cd server && pnpm typecheck` passes. - Query cost, local embedded Postgres, one 10-task request: about 49 SQL statements per task, 55–130 ms in total. A test fails if it goes above 60 per task. ## Risks - Database load: the stacked UI preloads up to 30 tasks, about 1,500 statements at idle time. This is the same work as opening those tasks, but it is done before the person clicks. The work runs in sequential requests of 10, with 2 tasks at a time. Most of the cost is existing run decoration. Live runs and the active run decorate the same running run, so that work runs twice. Sharing it is a follow-up. - Stricter access: live runs and the active run now apply the `issue:read` policy inside this endpoint. The existing routes keep their company-only check. - Hidden tasks are returned, as `GET /issues/:id` returns them. - No migration and no persisted-state change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code with tool use (shell, browser automation, file editing, subagents). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 15:05:41 -06:00
DottaandClaude Opus 5.5	653ab8b18f	fix(chat): answer "my inbox" from the person's Mine list (#20334 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agent Chat lets a person ask an agent about their own work, such as "what in my inbox needs my attention?" > - Agents find API routes through the agent API catalog, which is built from the OpenAPI registry > - The catalog described `GET /api/agents/me/inbox/mine` as the current agent's assigned items and listed no `userId` parameter. The skill reference for the route is keyed with `?userId={}`, so it never attached to the catalog entry > - A chat agent called the route without `userId`, got 400, and then answered from an unscoped company task search. It named company work that is not in the person's inbox and did not link the person's Mine list > - This pull request documents the route as the person's Mine list with a required `userId`, and adds one chat guidance paragraph for inbox questions > - The benefit is that inbox answers use the person's own task scope and link their Mine list ## Linked Issues or Issue Description What happened A person asked a CEO chat agent: "What in my inbox needs my attention today?" The agent called `GET /api/agents/me/inbox/mine` without `userId` and got 400. Then it listed every company task, including an unrelated agent-created maintenance task. It said the person's inbox was board-only and did not link the Mine list. Expected behavior The agent reads the person's Mine list with `GET /api/agents/me/inbox/mine?userId=<person>`. It answers from those tasks only, links each task it cites, and links the Mine list at `/<prefix>/inbox/mine`. If the read fails, it says so and links Mine. Steps to reproduce 1. As a board user, participate in some tasks. Let an agent create an unrelated task in the same company. 2. In Agent Chat with the CEO, ask "What in my inbox needs my attention today?" 3. The agent calls the Mine route without `userId` (400) and lists all company tasks. ## What Changed - `server/src/routes/openapi.ts`: `GET /api/agents/me/inbox/mine` now has a summary and description that say it is a person's Mine list, not the agent's own queue. It documents the required `userId` query parameter and the optional `status` parameter, and the 400 response. - `server/src/services/agent-conversations.ts`: one chat guidance paragraph. For inbox questions, read the Mine route with the `authorUserId` of the person's message, answer from those tasks only, and link `[Mine](/<prefix>/inbox/mine)` (it opens the Mine view in every layout). If the read fails, say so and link Mine. - Regression tests in `openapi-routes.test.ts` and `agent-conversations.test.ts`. ## Verification - `npx vitest run src/__tests__/openapi-routes.test.ts src/__tests__/agent-conversations.test.ts src/__tests__/heartbeat-list.test.ts` (in `server/`): 107 passed. The two new tests fail without the source change. - `npx vitest run src/services/native-runtime/runner-api.test.ts`: 2075 passed. - Live chat eval `inbox-search.show-my-work` on the eval candidate with this commit: see the comment below for the run result. ## Risks - Low risk. The change is catalog text, one query schema in the OpenAPI document, and one chat guidance paragraph. The route behavior does not change. - The chat prompt grows by about 500 characters. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 14:54:24 -06:00
Devin FoleyandClaude Opus 5.5	90ec0b9673	fix(sentry): report a reviewed chat setup wait only after its retry decision (#20205 ) A reviewed chat attestation that exhausts its short lock or delivery wait fails setup with reviewed_chat_execution_binding_not_ready, and the bounded retry then owns recovery. The failed-status write sent the Sentry report before the retry was scheduled, so a wait that the retry recovered still created an unexpected-failure event. The setup path now marks that write as pending and reports once after the retry decision. A scheduled retry keeps the failure local; an exhausted, unscheduled or failed retry decision still reports. Other setup failures keep their report on the failed-status write. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 12:32:15 -08:00
DottaandClaude Opus 5.5	7f55323abb	fix(recovery): end a remote workspace epoch whose writer died during preparation (#20346 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A task with a remote workspace prepares that workspace before the run selects its runtime, and preparation marks the workspace epoch dirty with the run as its writer > - If the server is killed during preparation, the run never selected a runtime and never started provider work, but the dirty epoch remains > - Failed-workspace preservation, the path that resolves a dead writer's dirty epoch, applies to native runs only, so nothing owned this epoch > - The dirty epoch kept the dead run's sandbox lease as a pending export and held the workspace against the task's retry, so the task never recovered > - This pull request lets the ownership reconciler end such an epoch at the last committed save > - The benefit is that a restart during workspace preparation no longer wedges the task ## Linked Issues or Issue Description What happened? In a restart-chaos wave on Daytona (a remote workspace that clones a real repository), the controller was killed while each run's first workspace save was in flight. That save is now settled by #20326. Each run ended `process_lost` with a bounded retry, and both runs were still in `execution_stage: preparing` with `runtime_mode_resolved_at: null`. The workspace authority stayed `dirty: true`, `snapshotId: null`, with the dead run as writer. The dead run's environment lease was proven owner-gone but was refused by its settle as `workspace_export_pending`, and the reconciler reported `I2_settled` violations. The retries stayed `queued` until the test harness deleted the environment 21 minutes later. Then they failed `remote_workspace_execution_required`, because the environment was gone. Expected behavior When a writer dies before runtime selection, its unsaved files are only the restored source and preparation output, because no agent turn ran. Recovery must end that epoch at the last committed save, so the lease settles and the retry restores the workspace into a new directory. A writer that reached runtime selection must keep its existing recovery. Steps to reproduce 1. Run a task whose agent uses Paperclip Runner with a remote sandbox workspace. 2. Kill the server while the run is still preparing the workspace (for example during its first save). 3. After the restart, the run ends `process_lost` and its retry is queued, but the workspace stays dirty and owned by the dead run, the lease stays active, and the retry never starts. Paperclip version or commit `master` at `251c6d5c69` (after #20326). Refs #20326 (settles the save that was in flight when the controller died). ## What Changed - New `settlePreparationOnlyRemoteWorkspaceEpoch` (`recovery/remote-workspace-unsaved-files-settlement.ts`), wired as the settle of the existing `remote_workspace_unsaved_files` ownership kind. Once the writer run is proven gone, it ends the epoch with the existing `abandonUnverifiedRemoteWorkspaceEpoch` only when all of these are true: - the run is terminal; - its runtime was never resolved (`runtime_mode_resolved_at` is null); - it ended in `execution_stage: preparing`; - it has no native finalization; - durable records show no work began: no `adapter.invoke` or `adapter.progress` events, no `turn.`, `provider.` or `native.process_*` events, and no recorded process id, process group, native session or runner instance on the run; - the workspace authority is still dirty, with this run as writer; - no save for that epoch is still `running` or `pending`. The epoch then advances with no writer and no location, and the abandoned epoch is recorded in `remoteWorkspaceAbandonedEpochs` with an activity entry. - After that, the lease no longer matches a prepared source, so the existing environment lease settle releases it and the workspace writer hold is free for the retry. - The ownership inventory row for this kind names the new settle. ## Verification - Red before the fix, green after: `ends a dirty remote epoch whose writer died during preparation, and keeps every other writer's files` (`ownership-reconciliation-report.test.ts`). On master it settles 0 epochs. With the fix it ends only the preparation-only epoch, and keeps the dirty epoch of a writer whose runtime was resolved, a writer that reached execution, a live writer, and a writer with a save still in flight. - Red before the fix, green after, end to end: `lets a retry prepare the workspace again after its first writer was killed during preparation` (`remote-workspace-save-concurrency.test.ts`). It runs the real reserve and prepare. The first writer's initial save is cut off, which leaves a first epoch with no committed save (`snapshotId: null`). The run is reaped as `process_lost` while still preparing, with its lease left active. Two reconciliation passes then end the epoch and release the lease. The retry reserves the workspace and prepares it again from the project base in a new directory, then commits its own initial save. On master, the epoch stays dirty. - The safety cases also keep a preparation-stage writer with an `adapter.invoke` event, and one with a recorded process id. - Red before the fix, green after: `releases the lease and frees the workspace for the retry once a preparation-only epoch ends`. Before the passes, the lease counts as a prepared source and the workspace writer hold is `held` for a retry. After two reconciliation passes, the lease is no longer active, the prepared source is gone, and the writer hold is `free`. On master the prepared source remains. - The reconciler, retained-source fallback, remote workspace lifecycle, save-concurrency, warm-handoff recovery, and retained orphan sandbox suites pass (139 tests, 1 environment-gated file skipped). After rebasing onto #20326, the reconciler, save-concurrency, retained-source fallback, and controller instance lease suites pass (91 tests). - `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm typecheck` in `server` passed. - Evidence: the runs in the wave's kept database all had `runtime_mode_resolved_at: null` and `execution_stage: preparing`. Their `runtime_mode: legacy` is only the column default, not a resolved runtime. ## Risks - This ends unsaved files without saving them. That is safe only because no provider turn can have started before runtime selection, and the guard requires the writer never to have resolved a runtime. A writer that did resolve one keeps its existing preservation path. - The abandoned root stays in the sandbox and is not reused. The next preparation restores the last committed save, or the source, into a new directory. - No schema or API change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, local database inspection). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 14:12:51 -06:00
DottaandClaude Opus 5.5	8a6b082274	fix(api): send call_api mutations once when responseText asks only for the first window (#20344 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native agents use the `call_api` fallback tool for API operations that have no dedicated tool. > - `responseText` is an optional `call_api` argument. It returns a bounded text window of a GET response. > - The server rejects every `responseText` on a mutation, also a first-window request that cannot repeat the mutation. > - A native agent attached `responseText` to a company rename PATCH. It got the same 400 on each retry, and no call succeeded. > - The same call also had no `body`. A PATCH without a body changes nothing, but the route returns 200. With the first rejection removed, the agent could read that 200 as a completed rename. > - This pull request sends a mutation once when `responseText` asks only for the first window, and it rejects a PATCH without a body before dispatch. > - The benefit is that a valid mutation goes through on the first call, a later-page request still cannot repeat a mutation, and an empty PATCH cannot look like a successful update. ## Linked Issues or Issue Description Refs #18105. That open pull request keeps the 400 for all mutation `responseText` values and corrects its message. This pull request changes the same check. It accepts a first-window request and keeps a corrected 400 for a later page. Whichever pull request merges second must resolve this line. What happened A chat agent (OpenCode, `deepseek-v4.1-flash`) tried to rename its company with `call_api`. We recovered the exact arguments from the payload digest in the run events: `operationId: "PATCH /api/companies/{companyId}"`, `pathParams.companyId`, `query: {}` and `responseText: { limitBytes: 8192 }`. There was no `body`. The OpenCode MCP bridge validates the arguments with Ajv without `useDefaults`, and `8192` is not a default in any schema. Thus the model, not the bridge, sent `responseText`. The server returned "responseText requires a GET operation". The agent sent the identical call 10 more times and one identical branding PATCH. No call succeeded. A second chat case in the same cohort sent `responseText` on `POST /api/issues/{id}/inbox-archive` three times before it recovered. Expected behavior `responseText` with no offset or offset 0 changes only how the response is shown. The mutation must run once and return its response on the normal inline or saved-artifact path. A later page (`offsetBytes > 0`) must still fail before dispatch, because it would repeat the mutation. A PATCH without a body must fail before dispatch with a message that names `body`, the allowed fields and the received argument keys. Steps to reproduce 1. Call `executeRunnerApi` with `PATCH /api/companies/{companyId}`, a body, and `responseText: { limitBytes: 8192 }`. 2. Before this change: a 400, and no HTTP request. After: one PATCH with the body, and the normal response. 3. Call it with no body. Before: the request is sent and the route returns 200 with no change. After: a 400 that names `body`, and no HTTP request. ## What Changed - `validateRunnerApiCall`: for a non-GET operation, drop a first-window `responseText` (no offset or offset 0) and continue. A later page still returns 400 with "This request was not sent" and the reason. - New `assertRunnerApiDispatchable`: reject a new PATCH with a required request body when `body` and `files` are both absent. The message lists up to 12 allowed fields and the received argument keys. `call_api` runs it inside its receipt reservation, after saved and inherited receipts are read and before it writes a pending receipt, so a rejected call leaves no receipt. `executeRunnerApi` also runs it for direct callers. Recovery and receipt replay use only `validateRunnerApiCall`, so a body-less PATCH that an older server completed still returns its saved result. - Tests: first-window `responseText` on a PATCH dispatches once with the body (three window shapes). A later page is rejected with no dispatch. A body-less PATCH is rejected with no dispatch, and `body: {}` is still accepted. The catalog-wide dispatch tests now send `body: {}` for PATCH operations with a required body. ## Verification - Red: the 5 new or changed cases failed on `origin/master` (`runner-api.test.ts`). - Green: `npx vitest run src/services/native-runtime/runner-api.test.ts src/services/native-runtime/runner-tool-results.test.ts` passed (2,087 tests). `tests/runner-e2e/api-response-reading.test.ts` passed (14 tests). - `tsc --noEmit -p server/tsconfig.json`: no new errors. The error count is the same before and after this change; the remaining errors come from packages that are not built in this worktree. - Upgrade safety and receipts: a new integration test seeds a completed receipt for a body-less PATCH. `recover()` and a same-callId `execute()` return the saved result. A new body-less PATCH is rejected twice with the same error, writes no receipt and sends no HTTP request, and the corrected PATCH with that callId then succeeds. The test failed with the rule in `validateRunnerApiCall`, and again with the rule after the reservation; it passes now. `runner-api.integration.test.ts`, `runner-api.test.ts` and `native-case-tools.integration.test.ts` pass (2,164 passed, 2 skipped). - One paid chat eval run (GitHub Actions run 42680925312) with the first commit cherry-picked onto the eval branch: `companies.rename-company` passed. The agent sent one correct PATCH with a body, attributed to the agent. That run did not send `responseText`, so it does not exercise the changed path; the unit tests cover it. ## Risks - Behavior change: a mutation with a first-window `responseText` now runs. Before, it failed. The response comes back on the normal path (inline up to 24 KiB, else a saved artifact), not as a text window. - Behavior change: a caller that sent a PATCH with no body to a route with a required body now gets a 400. Such a call changed nothing before. POST, PUT and DELETE are not changed, because some of them have useful empty bodies. - No route, permission or company check changes. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 14:03:08 -06:00
DottaandClaude Opus 5.5	93f7c3b5d3	fix(tasks): keep a budget-held Retry's failure visible and refresh the open task (#20338 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a task's automatic recovery stops, the person's Retry rechecks the task's gates. If a gate refuses, for example a budget hard stop, the Retry is saved and starts by itself once the gate allows it > - After a person pressed Retry under a budget hold, the task page went blank: no failure, no notice and no Retry. A person had no way to see why nothing started > - Two gaps caused it. The task notice drew only system-owned states, and a budget hold presents as a person-owned state with no card of its own. Also, admission saved the wait reason without publishing anything that an open page listens to > - This pull request keeps the failure visible and names the budget wait, and it refreshes an open task when admission records why a Retry waits > - The benefit is that "failure remains visible if the new attempt cannot start" holds without the person reloading the page > - These two commits were pushed to #20306 after it was squash-merged, so they are not on master ## Linked Issues or Issue Description Refs #20306 (merged without these two commits) and #20270. What happened? In the chat eval `chat-features.runner-opencode.local.recovery.retry-exhausted`, a person pressed Retry on a task whose native recovery was exhausted while the agent's budget was held. The Retry was accepted (`POST /api/issues/{id}/retry` returned 202). The task page then stayed blank for more than 60 seconds on the open page. Expected behavior The task keeps the failure visible and says that it is waiting for the budget. Once the budget allows new work, the saved Retry starts once by itself. Steps to reproduce 1. Let a task's run fail, and put the assignee over a budget hard stop. 2. Open the task and press Retry. 3. The page shows nothing about the failure or the wait. Paperclip version or commit master at `76254ead22` ## What Changed - `ExecutionBlockerNotice.tsx`: a Retry held by a budget hard stop presents as `waiting_for_user` with `userAction: "budget"`. The notice now shows "Couldn't finish this attempt. Your message is saved. It is waiting for the budget to allow new work. Raise the budget in Costs." (`data-testid="task-execution-waiting"`). It shows no second Retry, because the saved Retry starts by itself. Other person-owned decisions, such as approvals and run-stopped confirmation, keep their own surfaces. - `deferred-execution-wait.ts`: when the wait reason of a deferred request changes, its task records an `issue.execution_waiting` activity. Its live event refreshes an open task page. Callers inside a transaction (`heartbeat/queue.ts`) pass their post-commit publication list, so the event is published only after commit. ## Verification - A probe on the real deferred-receipt fixture showed that a budget-held Retry presents as `waiting_for_user` / `budget` / `canRetry: false` at 0 and 11 seconds. Before this change, the notice rendered nothing for that state. - `ExecutionBlockerNotice.test.tsx`: 18 tests pass. The budget-held notice is covered, it offers no duplicate Retry, and approvals still render nothing here. - `deferred-execution-wait.test.ts` (new, embedded PostgreSQL): - One live event is published for each new wait reason. The same reason again publishes nothing. - Inside a caller's transaction, nothing is published before commit. - A request that is no longer deferred records nothing. - Two of these cases fail when the change is reverted. - `deferred-execution-receipt-work.test.ts`: passes (49 tests together with the new file). - `pnpm check:token-gates`: clean. - Paid live run: `recovery.retry-exhausted` on an eval branch that includes both commits. The run ID will be added below. ## Risks - Low. The new notice renders only for `waiting_for_user` with `userAction: "budget"`, which rendered nothing before. - The activity row is written only when a deferred request's wait reason changes, so a gate that refuses repeatedly adds one row, not one row for each evaluation. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 13:52:49 -06:00
DottaandClaude Opus 5.5	16e9157577	fix(api): publish agent-grant access operations as board_or_agent in the API catalog (#20349 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agents read the API catalog (`server/src/routes/openapi.ts`) to decide which operations they may call. The `x-paperclip-authorization` actor tells them who performs each operation. > - The invite, join-request and member-list routes admit agents that hold the matching company grant (`users:invite`, `joins:approve`, `users:manage_permissions`). > - The catalog still listed these operations as board-only. > - In a chat eval, a CEO agent with `users:invite` read `actor: "board"` on `POST /invites`, treated it as a final boundary, and told the person it could not invite anyone. > - This pull request publishes those operations as `board_or_agent` and names the grant each needs. It also names the Members page on member edits, which stay board-only. > - The benefit is that a granted agent uses the permission the board gave it, and agents without the grant still get a 403. ## Linked Issues or Issue Description What happened? A board granted the CEO agent `users:invite` with `PATCH /api/agents/:id/permission-grants`. The person asked in chat: "Invite alex@example.test to join this company." The agent searched the API catalog. It found `POST /api/companies/{companyId}/invites` with `x-paperclip-authorization: { actor: "board" }`. It concluded that invites are board-only and did not try the call. The route itself accepts agents with `users:invite`. Expected behavior The catalog shows `board_or_agent` for operations whose route accepts an agent with a company grant. The description names the grant. The agent creates the invite and returns the join link. Steps to reproduce 1. Grant an agent `users:invite`. 2. Read `GET /api/openapi.json` and find `POST /api/companies/{companyId}/invites`. 3. See `x-paperclip-authorization: { actor: "board" }`, although `assertCompanyPermission` admits the agent. Paperclip version or commit Master at the merge base of this branch. Deployment mode Any. Found in the runner full-stack chat eval `access.invite-teammate`. ## What Changed - Remove six operations from `BOARD_ONLY_OPERATIONS`: `POST`/`GET /companies/{companyId}/invites`, `GET /join-requests`, `POST /join-requests/{requestId}/approve` and `/reject`, and `GET /members`. Their routes use `assertCompanyPermission`, which admits agents with the grant. - Add descriptions that name the grant for each operation. The invite description also says it returns a shareable `inviteUrl` (no email field), and that membership starts only after acceptance. - Member edits (`PATCH /members/{memberId}`, `/role-and-grants`, `/permissions` and `POST /archive`) stay board-only. The route guard requires board access, and a test keeps the catalog in line with that guard. - Describe those member edits as board-only even for agents holding `users:manage_permissions`, and name the Members page (`/<prefix>/company/settings/members`) where a person makes the change. The member list description says the same. An agent then links the page instead of searching for an agent route. - Add a regression test in `openapi-routes.test.ts`. ## Verification - `cd server && node ../node_modules/vitest/vitest.mjs run --config vitest.config.ts src/__tests__/openapi-routes.test.ts` — 43 passed. The new test also checks that the Members page is named. With master's `openapi.ts`, the new test fails (`expected { actor: 'board' } to deeply equal { actor: 'board_or_agent' }`). - `src/services/native-runtime/runner-api.test.ts` — 2075 passed. ## Risks - Low risk. This changes catalog metadata only. Route guards do not change. An agent without the grant still gets 403. - Agents with these grants will now attempt these operations instead of stopping. That is the intended behavior of the grant. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 13:52:43 -06:00
DottaandPaperclip	c67909cb0f	fix: keep tool approvals valid through OAuth refresh and delayed review (#20340 ) ## Thinking Path > - Paperclip lets people manage agent work and approve app actions. > - An approval must bind the action that the person reviewed. > - OAuth access tokens can expire while that person is away. > - The old target snapshot treated each token refresh as a new authorization. It also imposed a one-hour review deadline. > - This pull request separates token renewal from authorization identity and keeps the human decision durable. > - Current access, the reviewed arguments, meaningful target changes, task closure, and execution receipts still control dispatch. > - People can approve later without repeating a decision after routine connection maintenance. ## Linked Issues or Issue Description What happened? A pending app action returned `approved_tool_target_changed` after its OAuth tokens refreshed. The tool and connection settings were unchanged. The same card also expired after one hour. A sign-in problem could discard an accepted decision before provider dispatch. Expected behavior The original card remains reviewable after routine OAuth refresh and a server restart. A later approval runs the exact reviewed action once. A connection repair preserves the decision while Paperclip verifies the same authorization. Meaningful changes require a new review. Steps to reproduce 1. Connect an OAuth MCP app and configure an action to require approval. 2. Request the action, then refresh its access and refresh tokens before approving. 3. Approve the original card. Before this change, it returns HTTP 409 even when the authorization is unchanged. 4. Wait more than one hour. Before this change, the card expires without a task-specific deadline. Paperclip version or commit Reproduced from master at `76254ead2229`. The final branch is rebased onto current master. ## What Changed - Add versioned approval target comparison. New snapshots bind credential identity epochs, consent, permissions, ownership, connection configuration, tool contracts, and credential references. Authenticated OAuth refresh preserves identity. Manual credential replacement does not. - Remove the implicit one-hour deadline from ordinary app action reviews. Keep explicit deadlines and terminal-task expiry. Synchronize the nullable interaction deadline in shared contracts and UI. - Keep accepted, undispatched actions when sign-in is required. Show a saved-approval repair state. Resume on committed connection changes after target and access revalidation, without an idle retry loop. - Preserve the existing execution claim, result receipt, and uncertain-outcome no-replay rules. - Explain meaningful target changes without inviting the person to retry the stale card. - Add a production task-thread Storybook example and document compatibility and recovery behavior. ## Verification - Red: production-refresher invalid-grant handling and formal-approval withdrawal during a sign-in wait also reproduced as failing regressions before their fixes. - Red: before implementation, the local OAuth HTTP fixture failed for both public and confidential clients with `approved_tool_target_changed` after token refresh. - Green: the extended fixture passes for public and confidential clients with personal and shared grants. It covers actual local HTTP token exchanges, multiple refreshes, a rebuilt gateway, an ended originating run, approval after 25 hours, duplicate approval, preserved sharing arguments, sign-in waiting, and event-driven repair. The gateways use the production OAuth refresher; an invalid refresh preserves the accepted decision before provider dispatch. Omitted-scope responses cover equivalent scope-reporting changes. The provider is an isolated local fixture, not a deployed Notebook service. - Negative controls refuse changed consent, scopes, tool definition, destination, and manual credential replacement before provider execution. - Existing gateway service, connected-tool acceptance, secret owner, secret service, unsigned-review queue, and connection-health suites passed. Explicit-deadline fixtures still prove expiry behavior, including deadlines on accepted sign-in waits. A withdrawn formal approval cancels a waiting action before connection recovery. - Approval card, task-chat, error-copy and target-comparison tests passed. Server and UI typechecks passed. Token gates and Storybook catalog checks passed. - Inspected the new saved-approval repair card in local Storybook using the production task thread. It shows a reconnect link and no second approval button or false running state. - `pnpm verify --explain` returned exit 2 and requested broad CI coverage. [Full CI](https://github.com/paperclipai/paperclip/actions/runs/42677855507) passed on `eb4f1c7caa9ee6e802c314ffc5eadec475a5de0c`, including build, typecheck/release registry, general and serialized tests, and all eight browser shards. - Greptile reviewed the same final head at 5/5. Both findings from the first review are fixed and their threads resolved; no open follow-ups remain. ## Risks - No database migration is needed. The new comparison reuses the vault identity epoch. Only authenticated OAuth refresh code can preserve that epoch; HTTP credential-write schemas cannot select that behavior. - Earlier signed requests keep their original exact-version checks and saved deadlines. Already-expired cards are not revived. A historical card whose credentials already changed needs a fresh review. - A new consent, account, permission set, destination, or manually replaced credential must not inherit the old authorization. Reconnection continues automatically only when revalidation proves the same saved authorization. - The interaction enrichment now permits a null deadline. Existing string deadlines remain valid. The web client and shared validator change together. - This PR does not deploy to staging or perform a live Notebook publication. ## Model Used OpenAI Codex, GPT-6 family, with repository tools, code execution, and browser inspection. The runtime does not expose the exact model ID, context-window limit, or reasoning-mode label; those values are not asserted. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-09 13:46:28 -06:00
DottaandPaperclip	0e325e8d52	fix(runner): settle saved handoffs after verified shutdown (#20336 ) ## Thinking Path > - Paperclip manages AI agents and their work. > - An agent can save a question or approval handoff before its provider stops. > - A missing provider terminal notification does not prove that the saved work failed. > - Runner already verifies provider exit, event drain, tool settlement and suspension. > - This change uses that proof to settle the same saved handoff. > - Unproven shutdown holds continuation and explains that the work is saved. > - Recovery must never repeat the completed work or allow overlapping execution. ## Linked Issues or Issue Description Refs #20330. That PR extends the Codex terminal drain window. This change handles the separate case where no terminal arrives but the owned shutdown can prove stopped execution. What happened? A native run saved a human handoff, accepted cancellation, and later stopped its provider. The missing terminal notification still made the attempt fail and show “Couldn’t finish this attempt.” Expected behavior Keep the saved handoff. Accept its yielded result after either an authoritative provider terminal or verified shutdown of the exact execution. If shutdown remains uncertain, retain the handoff and explain the hold without offering a duplicate Retry. Steps to reproduce Save a governed wait, acknowledge interruption, and omit the provider terminal. The new regression uses the real Runner binary and a controlled Codex process. The old runtime rejects the handoff instead of asking the existing shutdown owner to settle it. Deployment mode Native Runner with Codex. Local verification used macOS arm64 and Node 26.4.0. Hosted sandbox recovery has not been qualified. ## What Changed - Add an optional driver shutdown-settlement operation. Runner returns proof only after its existing stop, drain, tool settlement and suspension barrier passes. - Bind the receipt to the run, normalized session and turn. Persist it with the saved wait and validate it during recovery. - Complete the same yielded handoff after verified shutdown. Retire the stopped session from warm reuse. Do not invent a provider terminal or complete usage. - Retain a named shutdown-pending condition under the existing bounded same-run recovery policy. Preserve actual provider failures and integrity failures. - Show saved-work shutdown guidance in task and inbox presentation. Keep genuine approvals authoritative and disable duplicate Retry. - Document the completion and restart contract. The checkpoint field is additive; older checkpoints retain the terminal-or-shutdown path. ## Verification - Red: the new runtime regression failed against the original implementation because verified shutdown was never requested. Server presentation and error-classification regressions also failed before the fix. - Red: both inbox shutdown states showed generic text before the inbox change. - Green: Runner runtime, shutdown and harness-backend suites pass 301 tests. Tests cover missing terminals, failed stop proof, wrong identity, cancellation, late proof, genuine provider failure, and restart after receipt persistence. Recovery completes once without another model turn. - Green: the native executor and task-presentation suites pass 1,371 tests. Execution projection, inbox and blocker presentation pass 125 tests. Two test-authoring mistakes were corrected: an integrity-error text expectation and an assertion for a nonexistent projection field. Their initial failures remain in local logs. - Runner TypeScript and full server layered type checks pass. Token gates and diff whitespace checks pass. - Live provider acceptance: real Runner and Codex with gpt-6.1-sol, one disposable local workspace, one tool call, and one deliberately dropped terminal notification. The saved file, bound settlement receipt, one accepted yielded result, one provider turn, confirmed provider exit and suspended Runner all passed. The tool handler and persistence port were controlled. This does not claim a full server or hosted journey. - The full browser question-and-answer cell stopped before execution because OPENAI_API_KEY was absent. No model work ran in that attempt. Hosted sandbox recovery and the full browser continuation journey remain unverified. - `pnpm verify` returned exit 2 because the execution-spec change requires broad CI. Required CI will supply that coverage; a full local repository suite was not repeated. - Review correction in `8553da244e`: a failed trailing-event write could hide a provider failure. A new regression reproduced red, then passed after all ordinary drain errors retained the named pending condition. No settlement receipt or result can be accepted after that error. The live missing-terminal check passed again on the corrected runtime. - The real PostgreSQL control-plane regression and three existing governed-wait controls pass. The new case persists one accepted yielded result, a bound shutdown receipt, and the original pending human question without a provider terminal. - First CI revision `7c2cb9b1`: test-program typecheck found a missing `PrpEvent` annotation in the new fixture. The annotation is fixed and the Runner test-program typecheck passes. Runner support also timed out in the unchanged `registered roots remain protected even when equal to genuine public static bytes` case after 30 seconds. One unchanged local diagnostic passed in 3.69 seconds overall. The hosted timeout cause remains unknown; no timeout, fixture, or assertion was weakened. [Original failed support job](https://github.com/paperclipai/paperclip/actions/runs/42670851821/job/129346900133). - Rebased onto the merged #20330 and current recovery changes. The combined branch passes 1,402 executor/presentation tests. Both prompt and 18-second provider terminals still pass. A warm session with no terminal passes after the full 30-second drain and verified stop/suspension. Its initial overly broad assertion rejected a nonterminal turn event; the corrected assertion rejects every terminal type. The rebased CI gate was cancelled before any runner or step started; its cause is unconfirmed and it supplies no validation credit. Final revision `b49ec5a144` starts fresh CI and review. - Final-head CI attempt 1 failed before creating jobs. GitHub reports an internal server error (correlation `2c576b07-b67f-4179-a69f-74f7d99cd926`), followed by empty runner-selection outputs for the aggregate jobs. One unchanged-commit retry was requested; the initial admission failure remains recorded. [CI run](https://github.com/paperclipai/paperclip/actions/runs/42673159868). - Current blocker: final head `b49ec5a144e94e41e96e2508a79bd9aec91fe45a` has a fresh 5/5 Greptile review and zero unresolved threads, but required CI is not complete. GitHub reports an active [Actions and Checks API incident](https://www.githubstatus.com/incidents/tp8cghg7tlc4). Attempt 2 ran only part of the full selected graph. Build, canary, Rust, sandbox providers, admitted browser shards and other completed test jobs passed. The E2E aggregate correctly failed because `policy` and `e2e_shards` had `abandoned` results. Server shard 5/12 received a runner shutdown signal at 18:36:55 UTC and its test step was cancelled; its job still reports in progress. The shutdown cause is not established. The separate review workflow had the same pre-execution internal-error class and passed one bounded retry. No further retry was submitted during the incident. Resume with the missing/cancelled coverage after GitHub recovers; do not count the absent typecheck, Runner or support jobs as passes. [Failed coverage gate](https://github.com/paperclipai/paperclip/actions/runs/42673159868/job/129354318236), [interrupted server shard](https://github.com/paperclipai/paperclip/actions/runs/42673159868/job/129350854174). ## Risks - A missing terminal can use one additional bounded shutdown interval. Unknown shutdown still blocks continuation. - Provider terminal and usage accounting keep their separate meaning. The receipt proves stopped execution and settled tool outcomes, not model success. - Recovery rejects a foreign receipt, a later turn, an active turn with a settled receipt, and an actual failed provider terminal. - There is no database migration. Older checkpoint shapes remain readable. Drivers without this optional proof retain a named pending state. - This PR does not deploy the fix or repair an existing hosted task. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, repository tools and local code execution. The exact serving model ID and context window are not exposed in this session. Live test inference used gpt-6.1-sol. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-09 13:46:16 -06:00
DottaandPaperclip	a67b936885	fix(server): redact invitation bearer paths from HTTP logs (#20343 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - People join a company through an invitation link. The link carries a bearer token in its URL path. > - The HTTP logger removes query secrets and URL user information. It keeps the path. > - So invitation bearer tokens reach the HTTP log in the request URL, the log message, and the referrer header. > - PR #17848 fixed this, but it did not merge. Since then, #13742 changed the same redactor file. > - This pull request carries the #17848 commits onto current master. The original author is kept on both commits. > - The benefit is that invitation tokens do not stay in HTTP logs. The chat eval for membership management can then accept a real invitation safely. ## Linked Issues or Issue Description Supersedes #17848 (same commits, rebased onto current master after #13742). What happened? The HTTP logger writes invitation bearer tokens to logs. A synthetic POST to `/api/invites/<token>/accept` puts the token in `req.url`, in the log message, and in the `Referer`/`Referrer` header record. Expected behavior Logs keep the invitation route, the method, and the HTTP status. Logs do not keep the bearer token. Steps to reproduce 1. Check out master. 2. Run `server/src/__tests__/logger-invitation-redaction.test.ts` from this PR. 3. The synthetic token appears in the request URL, the message, and the referrer. 12 tests fail on master. Paperclip version or commit Master at the merge base of this branch. Deployment mode Source server. The tests use a disposable local HTTP listener. They use no database and no provider. ## What Changed - Remove the bearer segment from `/api/invites/<token>/...`, `/invites/<token>/...`, and `/invite/<token>` URLs, including mounted paths and absolute URLs. - Apply the same URL policy to `referer` and `referrer` request headers. - Keep record IDs on the non-token revoke route `/invites/<inviteId>/revoke` and on company-scoped invite routes. - Add regression tests through the real HTTP logger for 202, 400 and 500 responses, with both referrer header spellings. - Original work by @cryppadotta in #17848. This PR only rebases it onto master. ## Verification - From `server`: `node ../node_modules/vitest/vitest.mjs run --config vitest.config.ts src/__tests__/logger-invitation-redaction.test.ts src/__tests__/redact-sensitive.test.ts src/__tests__/http-log-policy.test.ts` — 46 passed. - With the master `logger.ts` and `redact-sensitive.ts`, the same new tests fail: 12 failed. - `http-log-aborted-response`, `cloud-principal-log` and `setup-token-session` tests pass locally. Some other HTTP-log suites could not load locally because workspace package builds were missing in this worktree. CI covers them. ## Risks - Low risk. Referrer query and fragment values are now removed from request logs, as they already are for request URLs. This removes some diagnostic detail to protect secrets. - A route that uses `/invites/<id>/...` for a non-bearer record ID, other than `revoke`, now shows `[REDACTED]` in logs for that segment. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code execution, used to rebase and verify. The original change was authored in #17848. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-09 13:45:00 -06:00
DottaandClaude Opus 5.5	0f71ab796e	fix(issues): name the existing order in dependency-cycle denials (#20342 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Tasks can wait for other tasks through blocking dependencies, and the server rejects a dependency that closes a cycle > - The rejection said only "Blocking relations cannot contain cycles" > - In chat, an agent asked to "make Website wait for Design too" got this 422 while Design already waited for Website > - The agent had no sign that the existing order must stay, so it guessed a swap: it removed Design's dependency on Website and added the reverse > - This pull request makes the 422 name the existing order, say that nothing changed, and say to ask the requester which order they want > - The benefit is that agents keep the existing prerequisite and ask the person, instead of silently reversing it ## Linked Issues or Issue Description What happened A chat agent was asked to make Website also wait for Design. Design already waited for Website. The server correctly returned 422 "Blocking relations cannot contain cycles". The agent then removed the existing Design-waits-for-Website dependency and added Website-waits-for-Design. The person lost the existing prerequisite without being asked. Expected behavior The rejection keeps both saved dependency sets as they were. The agent explains the conflict with the real task names and asks the person which order they want before it changes any dependency. Steps to reproduce 1. Create tasks Website and Design. Make Design blocked by Website. 2. PATCH Website with `blockedByIssueIds: [Design]`. 3. See that the 422 says only "Blocking relations cannot contain cycles", with no task names and no next step. ## What Changed - `server/src/services/issues.ts`: the cycle check now records the path it finds. The 422 message names the existing chain by task identifier (for example "COM-2 waits for COM-1, so COM-1 cannot also wait for COM-2"). It never includes titles, because the chain can pass through tasks the caller cannot read. Identifiers are the same fields that locked relation stubs already show. It says no dependencies changed and the existing ones are kept. It says not to remove or reverse an existing dependency, and to ask the requester which order they want. - The 422 `details` now carry `code: "blocking_cycle"`, the updated issue, the requested blocker, and the existing chain as ids and identifiers. - `server/src/__tests__/issues-service.test.ts`: a regression test checks the message and details for a direct cycle, the chain text for a two-step cycle through a task with a private title, that no title appears in the error, and that the existing dependency is unchanged. ## Verification - `cd server && pnpm exec vitest run src/__tests__/issues-service.test.ts` passes (152 tests). The new test fails on the old message. - Chat eval `task-coordination.reject-dependency-cycle` ran end to end with the first commit (run 42678089404, which named tasks by identifier and title). The agent kept Design waiting for Website and asked the person which order to keep with a question card. The saved dependency sets did not change. The second commit removes titles from the message; the rest of the text is the same. ## Risks - Low risk. The status code (422) and the first words of the message do not change. Clients that match the full old message exactly would no longer match. No code in this repository does that. - The check reads the identifiers of the issues in the chain only when it rejects a request, so the success path is unchanged. - The error can show the identifier of a task that the caller cannot read. Relation responses already show this identifier in locked stubs, so this shows no new data. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 13:32:18 -06:00
DottaandClaude Opus 5.5	251c6d5c69	fix(recovery): settle a remote workspace save whose process died mid-save (#20326 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs in remote sandboxes save the workspace as snapshot operations (`workspace_operations`, phase `remote_workspace_snapshot`). Each save invocation is listed in `saveWork.activeIds` while it runs > - When a controller process is killed during a save, its invocation stays listed and the operation stays `running` > - Nothing recorded which process ran the invocation, so the shared ownership reconciler could not prove the owner gone and never settled the save > - The task's bounded retry then waited behind that save forever > - This pull request records the owning process for each save invocation, and lets the reconciler settle a save whose owner is proven gone > - The benefit is that a restart during a save no longer wedges the task ## Linked Issues or Issue Description What happened? In a restart-chaos wave on Daytona with a real cloned repository, two tasks each started their first remote snapshot save (`boundary: initial`, epoch 1). A SIGKILL hit the controller during the save, before any upload. Each run ended `process_lost` and got a bounded retry. The retries stayed `queued` forever, the snapshot operations stayed `running` with `saveWork.activeIds: [<dead invocation>]` and `quiescedAt: null`, and the sandbox leases stayed active. The ownership reconciler reported the saves as owner-gone but had no way to settle them. Expected behavior After a restart, recovery must settle a remote snapshot save whose owning process is gone. It is marked failed with provenance, so the task's retry can continue. A save that is still running must stay untouched. Steps to reproduce 1. Run a native task with a remote workspace (for example Daytona with a `git_repo` project source). 2. Kill the controller (SIGKILL) while the first workspace save is in flight, before its upload. 3. After the restart, the run ends `process_lost` and its retry is promoted to `queued`, but the save operation stays `running` and the retry never starts. Paperclip version or commit `master` at `1b7fe33432`. Refs #20291 (the quiesced case: a save with no active invocation) and #20233 (the same ownership pattern for local workspace commands). ## What Changed - `saveRemoteWorkspace` records the controller process stamp of each invocation in `saveWork.owners` when it registers the invocation. It removes that entry when the invocation settles. A module-level set marks invocations that are still running in this process (`isRemoteWorkspaceSaveInvocationLive`). - The ownership registry owns a `remote_snapshot_save` with active invocations by that stamp. A save whose active invocations have no stamps was written before this change and is owned by its run, as with finalize attempts. A mix of stamps, or more than one process, is unprovable. - `settleAbandonedRemoteSnapshotSave` settles a save with active invocations once its owner is proven gone: - Process-owned: the proven stamp must still match every active invocation, and none may be live in this process. - Legacy run-owned: the run must be terminal. - Either way it clears the active invocations, records them as `abandonedInvocationIds`, and marks the save `failed` with `lastFailure: remote_workspace_save_abandoned`. The #20291 re-arm of blocked fallbacks still applies. - The controller instance lease counts running saves stamped with this boot as held work, so a long save keeps its owner provably alive. - The ownership inventory row for this kind describes the stamp. ## Verification - Red before the fix, green after: `settles a remote snapshot save whose process died mid-save, and keeps live, foreign or unprovable ones` (`ownership-reconciliation-report.test.ts`). On master it settles 0 rows. With the fix it settles exactly two: - a save stamped by a dead process (proof `pid_dead`); - the wave's shape: an unstamped active invocation on a `process_lost` run. It keeps five kinds of save running: a save stamped by this live process, a stamp from another PID space, a legacy save on a live run, a save with mixed stamps, and (in the older #20291 test) a stamped active invocation. - `stamps each save invocation with the process that runs it and clears the stamp when it settles` (`remote-workspace-save-concurrency.test.ts`) pauses a real save mid-flight. It checks the owner stamp and the in-process live mark, then checks that both are cleared after the save finishes. - The reconciler, save-concurrency, remote workspace lifecycle, snapshot retention, retained-source fallback, preparation gates, and controller instance lease suites pass (153 tests, 1 environment-gated file skipped). - `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm typecheck` in `server` passed. - Not yet verified: the full Daytona wave replay that shows the queued retry starts after the save settles. I asked the chaos harness owner to replay it on this build. ## Risks - A legacy save (written before this change) with an active invocation is settled when its run is terminal. The finalize-attempt kind uses the same rule. During a rolling deploy, an old-release process could still be running such a save for a terminal run. If that save later succeeds, its publication still marks the operation `succeeded`. - If the retry in the wave is also held by the dead writer's dirty workspace authority, settling the save may not be enough to start it. The replay will show whether a second change is needed. - No schema or API change. `saveWork.owners` is new optional metadata. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 13:27:44 -06:00
Dotta	0eacad0d0c	fix: refresh preserved harness receipt facts without copying archives (#20322 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Volume-free deployment must preserve historical controller files before removing the old disk. > - A WEB preservation receipt binds an archive and source files to the selected session and database facts. > - Later metadata changes can make those facts stale while the preserved files and archive remain unchanged. > - The existing owner refuses every stale receipt, so preservation cannot converge. > - This pull request refreshes only facts freshness after it verifies the same source bytes, checkpoints and archive. > - Existing locks and an old-receipt digest comparison prevent a concurrent replacement from being overwritten. ## Linked Issues or Issue Description Refs #19814 for WEB evidence preservation without settling remote recovery. Refs #19832 for bounded larger inventories. Related harness receipt PRs were searched; no existing change provides this guarded refresh. What happened? A valid WEB preservation receipt becomes stale after later run metadata changes. Its original-volume owner refuses another preservation attempt because an invalid receipt already exists. This leaves an eligibility blocker even when the source and saved archive are unchanged. Expected behavior The original-volume owner can refresh stale facts only after it proves every other persisted binding unchanged. It retains the prior receipt and remote recovery state. Steps to reproduce 1. Preserve a settled session's WEB files and controller evidence. 2. Add later run metadata so its database facts digest changes. 3. Preserve the same scope again. The previous implementation refuses at the existing-receipt check. Paperclip version or commit The failure was reproduced on master `12523fb677585fcd6da929f868014514a4e96e87`. This branch includes master `1b7fe33432016b3b27ccdfbf7b8944da4b40726d`. Deployment mode Managed Linux server with PostgreSQL and durable object storage. ## What Changed - Reuse the exact prior archive for existing WEB receipts. Recheck current source bytes, controller history, checkpoint material and archive bindings. Read back ciphertext and verify its size and hash. - Permit only database/local-observation freshness changes. Selected session, source run, controller, archive, WEB entries, and assets must still match the prior proof. Local inode and timestamp observations may change across attempts; preserved bytes, paths and modes must match. - Recheck current facts and source files under the existing finalizer, task and run fences. Accept a matching concurrent winner or require the exact previous receipt digest before replacement. - Retain the prior database and local facts digests, witness fingerprint, observation time and receipt digest in the same company-scoped activity transaction. Require those four fields to reconstruct the exact prior proof from unchanged current bindings before replacement. Normal audit redaction stays enabled. - Preserve current-key checks and archive reads on ordinary idempotent calls. Keep active-source, changed-evidence and cancellation refusals. - Document the refresh contract. No schema, saved-run format or public API change. ## Verification - Genuine RED: the new real PostgreSQL/encrypted-archive case failed at the old previous-invalid gate. - Before the audit review follow-up, the full local owner suite passed 94/94 integration cases in 50.01 seconds. It covers same-archive refresh with no PUT, audit history, current-key failures, unreadable archives, cancellation, changed files, replaced volume, active sources, matching concurrent winners and conflicting receipt replacement. - The cases also cover a source becoming active during an idempotent readback. Earlier 91- and 93-case runs preceded these final review guards and are not the final result. - Server typecheck, module boundaries and diff checks passed on the final production/test bytes before the clean master merge. Required CI supplies final-head breadth. Four narrow post-merge refresh/current-key/idempotent regressions passed in 5.70 seconds on `82e270f29fff98442c4184b96082f72f18a88721`. - The fixtures use a real disposable PostgreSQL database and actual local encrypted archives/storage. Controlled faults affect the stated storage or concurrency boundary. They do not prove a hosted refresh or archive plaintext cold restore. - The existing integration file now exceeds 1,000 lines. Keeping these cases with its private original-volume, controller, archive and PostgreSQL fixture avoids duplicating that setup. - Initial worktree preflight stalled on the global package-manager wrapper. Preflight completed with the existing pinned toolchain. No setup failure is counted as a product test pass. - Review found that activity redaction could change a full prior-receipt copy. A new regression enabled the real username censor and reproduced that mismatch. The fix stores only the four changed scalar fields and their original proof digest; a prepublication canonical reconstruction guard rejects dropped or unknown fields. - On the audit follow-up, 16 affected real PostgreSQL cases passed in 11.98 seconds; 80 unchanged cases were intentionally skipped. They include real redaction with a username-containing WEB path, reconstruction across two refreshes, unknown prior fields, CAS, active sources, current-key failures, unavailable archives and cancellation. Final server typecheck and boundaries passed. This was an intermediate revision; the final-head results appear below. - A second review found that comparing old inode/timestamp observations also broke ordinary valid retries. Three real cases reproduced valid-receipt touch/inode failures and stale-facts touch failure. The fix permits metadata freshness changes across attempts while keeping exact byte/path/mode/checkpoint/archive coverage and the existing locked within-attempt metadata fence. - Final local owner suite on the metadata follow-up: 102/102 integration cases passed in 44.04 seconds, including valid touch/new-inode retries, stale refresh with the same archive and no PUT, four-scalar audit reconstruction, changed bytes/mode/entries, and a touch during archive readback that must refuse. Server typecheck, module boundaries and diff checks passed. The final-head CI and review results appear below. - Final head `c81475c927d0862dcb8d2c2356dc41ab4a27e63f`: [PR workflow attempt 5](https://github.com/paperclipai/paperclip/actions/runs/42667287803/attempts/5) completed successfully at 19:24:28 UTC on 2026-11-09. All 51 jobs are accounted for: 49 passed and 2 were skipped. The actual [final ci / verify](https://github.com/paperclipai/paperclip/actions/runs/42667287803/job/129382061558) and [E2E aggregate](https://github.com/paperclipai/paperclip/actions/runs/42667287803/job/129381486342) passed. Typecheck, build and the selected test lanes passed. The exact-head rollup is successful with no pending or failed checks. - Attempt 5 resolved both trusted workflows to `3a2f71db31aa99633610dc241f55f34ac65124de`, after #20339 merged. Its routing gate passed and real jobs started. This verifies successful startup and completion of this rerun; it does not exercise the empty-runner fallback because routing succeeded. - Earlier CI failures on `82e270f29fff98442c4184b96082f72f18a88721` and attempts 1–4 of this final-head workflow remain nonpasses. Attempt 1 had interrupted leaf jobs and failed aggregates. Attempts 2 and 3 ended in workflow failure without a final verify job. Attempt 4 failed startup with zero jobs and empty-runner errors. Those failures are not counted as passes or labeled unrelated. - [Greptile](https://github.com/paperclipai/paperclip/pull/20322#issuecomment-6746395090) and [Ponytail](https://github.com/paperclipai/paperclip/pull/20322#issuecomment-6746327827) reviewed the unchanged final head at 5/5. Both historical review threads are resolved; no inline thread remains open. - Hosted application to an actual stale scope remains pending normal release delivery. ## Risks - Refresh must not bless different preservation evidence. The unchanged-binding checks, fresh file and checkpoint proof, exact archive reuse, source capability and locked receipt comparison remain mandatory. - A fresh capability has a new witness fingerprint and expiry. The audit retains the old fingerprint, database/local facts digests and observation time. All unchanged proof bindings remain on the current receipt. Their canonical reconstruction must match the old digest. The new capability is validated at entry and publication. - Readback verifies ciphertext size/hash and current-key checkpoint material. This change does not claim a new archive GCM/plaintext restore. - Archive identity and rows remain unchanged, so retention ownership does not change. Failed refreshes retain the previous receipt, original files and unresolved remote recovery obligations. - Older receipt shapes are not rewritten automatically. Normal validation and named preservation refusals remain in force. ## Model Used OpenAI Codex, GPT-6.1 Sol (`gpt-6.1-sol`), with reasoning, repository tools and code execution. Context-window size is not exposed by the runtime. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge	2026-11-09 13:26:27 -06:00
DottaandPaperclip	3a2f71db31	fix(ci): handle missing runner output and await import fixture readiness (#20339 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - CI checks must report real result failures after their setup phase. > - Required summaries must run even when the routing gate has no runner output. > - Process-recovery tests must start their checkpoint deadline after the child loads its modules. > - An empty runner prevents summaries from starting. Slow child startup can consume the checkpoint deadline. > - This pull request adds a fallback for the two summaries and a ready signal for the recovery fixture. > - Existing routing, checkpoint, key and recovery assertions remain in place. ## Linked Issues or Issue Description Refs #20322. Related open PRs #12512 and #12513 propose a broader runner contract and caller activation. This change does not adopt that contract or change normal routing. No current repair for an empty aggregate runner was found. What happened? The PR workflow can fail before starting any jobs when the routing gate has no runner output. GitHub reports an empty `runs-on` value for the `verify` and `e2e` summary jobs. Their `always()` conditions make this path possible when a dependency cannot run. A real workflow rerun showed both errors at lines 801 and 1784 of the trusted workflow. The first run of this PR also failed two process-recovery cases with no saved child checkpoints. The fixture starts its 15-second checkpoint poll before a new Node process imports the server modules. A controlled 16-second pre-import delay reproduces the same zero-checkpoint assertion. That evidence proves a deadline-boundary defect. The original CI run has no phase events, so it does not prove which stage caused those two CI failures. Expected behavior The summaries can start on `ubuntu-latest` when routing has no runner output. They must still fail when routing or required work did not succeed. The process fixture waits for a bounded child-ready signal before its unchanged checkpoint deadline. It must still prove checkpoint reuse, process termination and refusal under a changed master key. Steps to reproduce 1. Start or rerun PR CI with no runner output from the routing gate. 2. Let the two `always()` summary jobs evaluate their runner expressions. 3. Observe that the previous expressions produce an empty value and GitHub refuses the jobs. 4. For the fixture defect, temporarily delay the child before imports for 16 seconds. The previous fixture fails its 15-second checkpoint poll before the child is ready. The updated fixture passes with the same delay. The temporary delay is excluded from this PR. Paperclip version or commit Reproduced with the workflow from `f56b097fbd25607b3b84e418085ab42c2cbb758d`. The caller and both reusable workflow files are byte-identical between that revision and the affected PR head. This fix is based on current master `4a349ea44754f0b1fb8e5e04dc6cdcb05d1c3396`. Deployment mode GitHub Actions PR CI. ## What Changed - Add `ubuntu-latest` as the fallback only for `verify` and `e2e` when the routing runner is empty. - Extend the existing workflow contract test to require a usable fallback and execute each real summary script with a failed, cancelled or skipped gate and absent routing outputs. - Wait up to 30 seconds for the recovery child to finish imports and construct storage before starting the existing 15-second checkpoint poll. Keep the 30-second work abort, 60-second test timeout, and every kill, key, state and proof assertion. - Preserve normal runner selection, task selection, permissions and all result assertions. No production code, public command or product contract changes. ## Verification - The new local workflow contract failed before the two runner-expression changes. The existing summary scripts rejected failed routing while the test identified the missing fallback. - 148 focused tests passed across CI planning, routing and E2E sharding in 16.00 seconds. These execute the actual summary Node scripts and preserve failure and cancellation checks. - `actionlint -oneline -shellcheck= -pyflakes=` passed for the trusted workflow, routing workflow and caller. `git diff --check` passed. - The canonical worktree preflight completed with the pinned toolchain and shared build cache. - This test does not implement a GitHub expression evaluator. The real GitHub startup error establishes the absent-output failure; the source contract and actionlint validate the proposed expression. Hosted execution of this fallback remains to be verified after the trusted workflow reaches master. - The unchanged two process cases passed locally before the fixture change: 2 passed in 31.06 seconds. A controlled 16-second startup delay then reproduced the zero-checkpoint failure in 21.81 seconds. The ready handshake passed with the same delay in 37.11 seconds. The temporary delay and phase logging were removed. These are measured test durations, not a measurement of the original CI child startup. - The final full nested-verification fixture passed: 38/38 cases in 132.30 seconds, with disposable local PostgreSQL. This includes partial checkpoint reuse, changed-key refusal and all existing recovery controls. - Fresh final head `da35ea034bdee603c021c4eba5b0f78ee6ef52d9`: the [full PR workflow](https://github.com/paperclipai/paperclip/actions/runs/42678538191), [final ci / verify](https://github.com/paperclipai/paperclip/actions/runs/42678538191/job/129374227690) and [incremental qualification](https://github.com/paperclipai/paperclip/actions/runs/42678537930) passed. The [previously failing server lane](https://github.com/paperclipai/paperclip/actions/runs/42678538191/job/129366881302) passed. The final rollup has 61 successful checks, 6 expected skips, 2 neutral checks and no pending checks or failures. Snyk passed. - [Greptile](https://github.com/paperclipai/paperclip/pull/20339#issuecomment-6748154727) and [Ponytail](https://github.com/paperclipai/paperclip/pull/20339#issuecomment-6748242241) both reviewed that exact final head at 5/5. There are no inline review threads. Ponytail permits retaining the source check with actionlint, which is the validation used here. - The [first workflow run](https://github.com/paperclipai/paperclip/actions/runs/42674612619) failed the two cases described above; it is not counted as a pass. No CI retry was used to obtain the final result. CI uses the merged trusted workflow; its policy tests validate this proposed source file. ## Risks - A routing failure now produces a normal failed summary instead of an invalid runner expression. The unchanged assertions require successful routing, policy, valid scope and selected suite results. - Only lightweight summary jobs can fall back to GitHub-hosted capacity. Heavy jobs and authorization rules retain their existing route. - The new readiness allowance is bounded. A child that does not become ready, fails its work deadline, or cannot produce the expected checkpoint still fails. The original CI failure cause remains unproved beyond the observed zero-checkpoint assertions. - The fallback assertion is an exact source contract. A formatting change can require an assertion update. The separate actionlint check validates expression syntax; the text check does not emulate GitHub. - This does not repair a GitHub internal error or runner interruption. Those failures remain nonpasses. ## Model Used OpenAI Codex, GPT-6.1 Sol (`gpt-6.1-sol`), with reasoning, repository tools and code execution. Context-window size is not exposed by the runtime. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-09 13:14:16 -06:00
DottaandClaude Opus 5.5	bce7503a82	perf(workspaces): time a restore's sandbox upload and extraction separately (#20331 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Remote (sandbox) workspaces are restored from a durable snapshot before a later turn, so restore time delays that turn's first token > - #20280 added run-log timing for the storage steps of a save and a restore > - A Daytona wave with a Paperclip-sized workspace (~750 MB) spent 39.9 s preparing the workspace of a restored run. The storage steps (download, decrypt, verify, publish) took 2.4 s, and `workspace.snapshot.sandbox_restore` took 35.6 s > - The archive helper unpacks the same archive locally in 3.6 s, so most of that time is elsewhere, probably copying 750 MB into the sandbox. One span cannot say which > - This pull request splits that span into the upload into the sandbox and the extraction in it > - The benefit is that the next measurement shows which part to fix ## Linked Issues or Issue Description Refs #20280 What happened? A restored run on Daytona spent 35.6 s in `workspace.snapshot.sandbox_restore` (wave `w-20261109T170632-s9921`, R1 workload, 754,838,327-byte snapshot). The run log records only that total, so it cannot show whether the copy into the sandbox or the unpack inside it is slow. Steps to reproduce 1. Run a remote workspace task with a large repository on Daytona and let a later run restore it. 2. Read the restored run's `run.performance.span` events. 3. Only `workspace.snapshot.sandbox_restore` covers the time after the storage download. Expected behavior The run log separates the upload into the sandbox from the extraction there. ## What Changed - `remote-workspace-transport.ts` (`restoreRemoteWorkspaceDirectory`): the archive copy into the sandbox runs inside `workspace.snapshot.sandbox_upload`, and the in-sandbox restore command inside `workspace.snapshot.sandbox_extract`. Both are children of the existing `workspace.snapshot.sandbox_restore`. - `execution-preparation-timing.ts`: the two new phase names. A traced `measureRunPhase` scope also carries its preparation ancestor's run (`recordRunId`). Without it, with an OTLP endpoint set, a step nested inside another step (here, upload and extract inside `sandbox_restore`) never reached the run log. Execution attempts still record nothing. - `doc/run-log-events.md`: the two rows in the durable storage timing table. - No behavior change. This is a run-log change: the rows stay in the instance database. ## Verification - New test `times the restore's sandbox upload and extraction as separate run-log steps` (`remote-workspace-transport.test.ts`). It restores a real archive through the real helper, inside the production `sandbox_restore` step with an OTLP endpoint set, with a 60 ms pause in each step, then reads the drained run phases. It fails without the spans or without the scope fix. - New test `records a step nested inside another step when an OTLP endpoint is set` (`execution-preparation-phases.test.ts`). It fails without the scope fix. - `npx vitest run` on `execution-preparation-phases`, `execution-preparation-timing`, `execution-timing-context`, `database-operation-attribution`, `native-workspace-finalization-work`, `remote-workspace-transport` and `remote-workspace-snapshot-storage`: 256 passed. ## Risks - Low risk. Two more measured steps on the restore path; spans under 50 ms are not recorded. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`, Anthropic), 1M context window, tool use and code execution, via Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 12:56:25 -06:00
DottaandClaude Opus 5.5	4d2a77cf5a	perf(workspaces): verify saved Git objects without a single-threaded full fsck (#20329 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Remote (sandbox) workspaces are saved to durable storage as encrypted snapshots, and restored before a later turn > - Saves and restores are a reported source of slowness on staging, so #20280 added run-log timing for each step > - A Daytona storage wave with a Paperclip-sized workspace (R1, ~750 MB, 19.5k entries) showed each save taking 58–68 s. Storage was only 2–3 s of that; capture inside the sandbox was 44–47 s > - The first save runs while the first run prepares its workspace, so it delays the first token > - A local profile of the same capture found one cause: `git fsck --full` re-hashes the whole history on one thread on every save > - This pull request keeps the same object checks but verifies packs on several threads > - The benefit is about 10 s less per save on a Paperclip-sized repository, with the same corruption protection ## Linked Issues or Issue Description Refs #18990, #20280 What happened? Saving a remote workspace with a large Git history is slow. On Daytona (3 vCPU), a Paperclip clone's save took 58–68 s; capture was 44–47 s of it. The first save is on the first run's critical path: `heartbeat.prepare.workspace` took 90–110 s before the first turn. Steps to reproduce 1. Clone Paperclip (`git clone https://github.com/paperclipai/paperclip.git`, ~750 MB, one 459 MiB pack). 2. Run the generated archive helper: `node archive.cjs capture <clone> <out.pcws>` (from `remoteWorkspaceArchiveScript()`). 3. It takes ~30 s. A wrapper on `git` shows 22.5 s in one `git fsck --full --no-dangling` on the normalized copy. Expected behavior Capture verifies saved Git objects as #18990 requires (a missing or corrupt object blocks the save), without spending most of the save re-hashing unchanged history on one thread. ## What Changed - `remote-workspace-snapshot-archive.ts`: both `git fsck --full --no-dangling` calls (the offline source check and the normalized-copy check) now call a new `verifyGitObjects`. It runs: - `git fsck --connectivity-only --no-dangling`: every object reachable from refs, the index and saved Git state is present. - A direct hash of each loose object (inflate, then SHA-1 or SHA-256 per the repository's object format) against its file name. A mismatch or unreadable object fails with the existing `invalid_git_repository` reason. - `git index-pack --verify --threads=N` for every pack (N = available CPUs, at most 8). This re-hashes every packed object, on several threads. - `fsck --no-full` was not used: in a shallow clone it reports reflog entries that point at packed objects as invalid. - Tests (`remote-workspace-snapshot-archive.test.ts`): - New: capture refuses a repository missing a staged object, or a committed object. - New: capture refuses a repository with a corrupt byte inside a pack. - The existing #18990 test `rejects corrupt available objects in full/shallow/partial Git before publishing a receipt` (valid zlib, wrong hash) still passes. It failed with connectivity-only fsck alone, which is why the loose-object hash is there. ## Verification - `npx vitest run` on `remote-workspace-snapshot-archive`, `remote-workspace-transport`, `remote-workspace-migration`, `remote-workspace-online-source` and `remote-workspace-readonly-capture-budget`: 526 passed. - Local benchmark: a fresh clone of `https://github.com/paperclipai/paperclip.git` (753,506,675-byte archive, 19,506 entries), alternating runs on one host: \| Capture \| Run 1 \| Run 2 \| \|---\|---\|---\| \| master helper \| 32.04 s \| 30.05 s \| \| this branch \| 21.07 s \| 20.09 s \| Both produce the same archive: equal byte size, manifest digest (`e5facf033409…`) and sha256 (`a4eb843bff1f…`). - Component timings on the same clone: `fsck --full` 20.5 s; `index-pack --verify` 19.5 s on 1 thread, 10.7 s on 3 threads; `fsck --connectivity-only` 0.9 s. - Daytona evidence for the problem (wave `w-20261109T170632-s9921`, R1, 3 vCPU, MinIO): save `captureMs` 44–47 s, `transferMs` 11–20 s, storage `putMs` 1.3–1.8 s, `encryptMs` 0.4–0.7 s. I did not rerun Daytona on this branch: its image must match the checkout's runner content. The local helper run is the same code the sandbox executes. ## Risks - Same guarantees as before for missing and corrupt objects. `fsck --full` also applies its object syntax checks to every packed object; `index-pack --verify` checks hashes and pack integrity. Commits and trees are still parsed by the connectivity walk, so malformed ones still fail. - More CPU threads during capture, for a shorter time. Capture already runs inside the sandbox, off the control plane. - A follow-up could skip re-verifying a pack that an earlier save of the same workspace already verified. Most saves keep the same clone pack, so that would remove the remaining ~10 s on repeat saves. Not in this PR. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`, Anthropic), 1M context window, tool use and code execution, via Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 12:56:21 -06:00
DottaandClaude Opus 5.5	4a349ea447	fix(chat): tell agents that instruction saves keep no history (#20328 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Chat agents can read and edit other agents' instruction files with dedicated tools > - #14420 changed instruction saves: Paperclip now keeps only the current file and records no revision history > - The agent-facing descriptions still describe a "canonical instruction revision" history > - In a live eval, a chat agent asked to undo an instruction change found an empty history, read it as a test artifact, guessed the earlier text from the current file's wording, wrote it, and reported a restore > - This pull request tells agents the real contract in chat guidance and in the tool and API descriptions > - The benefit is that an undo with no saved version leaves the instructions alone and asks the person, instead of inventing an earlier version ## Linked Issues or Issue Description Refs #14420 Refs #19515 (open: would list save records from the activity log. Those records hold no content, so they still cannot restore earlier text) What happened A person asked a chat agent: "Undo Morgan's last instruction change." Morgan's history was empty, because saves do not record revisions. The agent's reasoning said the history "is only populated with canonical revisions" and that there was "no revision tracking in this e2e harness". The agent then removed the last line of AGENTS.md and replied "Undone. Morgan's AGENTS.md is back to …". No saved source had the earlier text, so the agent guessed it. Expected behavior When there is no saved earlier version, the agent leaves the instructions unchanged. It says plainly that no earlier version was saved, and it asks the person for the text they want or which part to remove. Steps to reproduce 1. Create an agent with an AGENTS.md file. Edit the file once through `PUT /api/agents/{id}/instructions-bundle/file`. 2. `GET /api/agents/{id}/instructions-bundle/history?path=AGENTS.md` returns `{"revisions":[]}`. 3. Ask a chat agent to undo the agent's last instruction change. The agent rewrites the file from a guess. ## What Changed - `AGENT_CHAT_DIRECTIVE` now says three things: - Saves keep only the current instruction files. - An empty history means no earlier text is retained. - An undo with no listed revision must leave the file unchanged, say that no earlier version was saved, and ask for the wanted text. - The `get_agent_instruction_history` contract now says that it lists only pre-upgrade revisions, and that an empty list means Paperclip retains no earlier version. - The `restore_agent_instructions` contract now says that only listed pre-upgrade revisions can be restored. - The OpenAPI summary for `GET /api/agents/{id}/instructions-bundle/history` says the same. - The semantic tool contracts, the protocol manifest and the seeded eval catalog hash are regenerated. - Regression tests check the new guidance and both live tool contracts. ## Verification - `pnpm exec vitest run server/src/__tests__/agent-conversations.test.ts server/src/__tests__/board-access-denial.test.ts server/src/__tests__/codex-local-execute.test.ts server/src/__tests__/runner-api.test.ts` passes. - In `packages/paperclip-runner`, `pnpm exec vitest run src/semantic-tools/paperclip-semantic-tools.test.ts src/catalog src/evals/native-execution.test.ts` passes. - `node scripts/generate-semantic-contracts.mjs --check`, `generate-protocol-manifest.mjs --check` and `generate-replay-goldens.mjs --check` pass. - `tsc --noEmit` passes for `server` and `packages/paperclip-runner`. - The live result is the chat eval `agents.restore-instructions`, run with this change cherry-picked: run 42666621831 passed. The agent replied that Morgan's history is empty and no earlier version exists. It left AGENTS.md unchanged and asked for the exact text or the line to remove. Run 42660236086, without this change, failed: the agent guessed and rewrote the file. ## Risks - Low risk. Only agent-facing text changes, and no behavior changes. - The runner semantic catalog hash changes. This is the same kind of change as #19983. - If #19515 lands, history lists save records without content. The guidance wording must then say that listed saves without content cannot be restored. Whichever PR merges second must reconcile the two texts. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 12:23:12 -06:00
DottaandClaude Opus 5.5	f56b097fbd	fix(recovery): move stranded legacy subtasks off a parent's held workspace (#20314 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents split work into subtasks. Since #19860, every subtask gets its own execution workspace from the project base, and a subtask that an older release bound to its parent's workspace moves to its own workspace on its next admission > - Some legacy subtasks never get a next admission. Another task held the shared workspace with unsaved files, so their runs failed setup before any provider work. Stranded-work recovery then blocked them and handed them to the board > - On staging, a parent and two of its children stayed blocked for four days this way. The children could have run in their own workspace at any time > - This pull request makes the scheduled recovery-action sweep move such a child to its own workspace and continue it, with no person > - The benefit is that a legacy subtask never stays blocked on a workspace that it should not share, and the parent keeps its own workspace ## Linked Issues or Issue Description Refs #19860 (subtasks get their own workspace; legacy children move at admission) Refs #20295 (the held workspace's exhausted save now settles automatically) Refs #20273 (stall wakes wait while a workspace hold refuses a run) What happened? A parent task and three children shared the parent's remote workspace, because an older release created the children before #19860. One child's ended run held the workspace with unsaved files. The other two children were woken, but each run failed setup with "The workspace still has an unfinished writer or save operation." (`setup_failed`, `executionRecovery.kind=bootstrap`, `providerWorkStarted=false`). Stranded-work recovery then set each child to Blocked with a board-owned escalation (`board_escalation_no_takeover_v1`) that asked a board operator to retry. The #19860 move happens only at the next admission, and a blocked child gets no admission. Expected behavior A subtask that failed only because of a legacy shared-workspace binding continues in its own workspace from the project base automatically. The parent and the shared workspace stay unchanged. Steps to reproduce 1. With a release before #19860, create a parent task and children that share its remote workspace. 2. Let one child's run end with unsaved files whose save is not settled. 3. Wake another child. Its run fails setup on the workspace hold, and stranded-work recovery blocks it with a board escalation. 4. See that the blocked child is never moved to its own workspace and never runs again. Root cause `server/src/services/execution-workspaces.ts` (`releaseAncestorWorkspaceBindingAtAdmission`) moves a legacy child only when a run of the child is admitted. `server/src/services/recovery/service.ts` (`reconcileStrandedAssignedIssues`) ignores Blocked tasks, so nothing admits a child that a setup failure on the shared hold blocked. ## What Changed - `server/src/services/execution-workspaces.ts`: extract the binding release into `releaseAncestorWorkspaceBinding(tx, …)`, which runs inside the caller's transaction. Admission uses it unchanged (same guards: no other running run of the task, no unsettled workspace recovery, only a relative's workspace). - New `server/src/services/recovery/legacy-workspace-rehome.ts` (`rehomeStrandedLegacyWorkspaceTasks`). It selects a board-owned stranded escalation whose latest run failed setup on the workspace writer hold before any provider work, for a Blocked task bound to a relative's workspace. In one transaction it releases the binding, resolves the escalation with the outcome `legacy_shared_workspace_rehomed`, sets the task to `todo`, records a pending wake (`legacyWorkspaceRehome.wake: "pending"`), and records `issue.legacy_workspace_rehomed` system activity. After commit it wakes the assignee once (`issue_recovery_action_restored`, with a fresh session) and marks the wake `delivered` only when a run is created. A wake lost to a crash, refused (`null`), or failed stays pending and backs off (5 minutes, doubling, at most an hour); the pass returns the earliest owed wake as its retry deadline, so the recovery worker schedules it. Candidates are paged by action id. - Guards: the Blocked status must come from that escalation (set after the failed run, and no status decision at or after the block), the shared dependency-readiness rule must allow work (cancelled blockers and done blockers that still save files keep it blocked), the failed run must belong to the current assignee, and the binding release must be allowed. Otherwise nothing changes. - `reconcileActiveRecoveryActions` (the company-scoped sweep that the scheduled issue-recovery worker runs for each company with an active recovery action) runs this pass first. `recoveryActionSweepCondition()` also selects a company whose moved task still owes its wake. - `doc/execution-semantics.md`: documents the move and what a moved child sees. ### What a moved child sees A fresh workspace from the project base, exactly like a new top-level task in the project. It does not get the parent's changes. It does not get its own earlier files from the shared workspace either, because nothing of its work ran there (its runs failed setup before any provider work). The child that still holds the shared workspace with unsaved files is not moved by this change: its save recovery owns the hold (#20295 settles it automatically), its files stay in the shared workspace and in that recovery's saved-work receipts, and it moves on its first admission after that recovery settles (the #19860 rule). ### Previous-release story (`rules/upgrade-safe-run-state.mdc`) - Migrated, no schema change. The change reads escalations and task bindings that older releases saved, and it changes only the task's binding row, the task status, and the escalation. It does not change saved run identity (native execution input, leases, checkpoints, sessions). The regression test seeds the exact staging shape (Nov 4–5 evidence fields, `retryReason: workspace_busy`, `latestRunErrorCode: setup_failed`). - Mid-run deploy. A child with a running run or an unsettled workspace recovery is not moved (the same guards as the admission release). ## Verification - Red (the move disabled), `npx vitest run src/__tests__/legacy-workspace-rehome.test.ts`: 1 failed, 2 passed. The staging-shape case failed because the stranded child stayed Blocked and bound to the parent's workspace. - Green: 8 of 8 pass. All cases run the scheduled path: the company is selected by `recoveryActionSweepCondition()` and the pass is `recoveryService(...).reconcileActiveRecoveryActions({ companyId })` with the real heartbeat wake. - Staging shape: a parent and 3 children share the parent's workspace. One child holds it with a dirty writer run. Two children are blocked by board escalations after setup failed on that hold. After the stranded-work sweep, both children are unbound from the shared workspace, their escalations are resolved by the system with text that asks no person to act, and each child is woken exactly once with a fresh session. Each child runs once through the real heartbeat path outside the shared workspace and completes. The parent and the writer child keep their bindings, the shared workspace row is unchanged, and nothing runs for them. A later pass changes nothing. - A move whose wake was lost to a crash: the scheduled worker still selects the company, delivers the wake once, and the child runs and completes. - A refused wake (`null`) or a thrown enqueue stays pending, backs off, does not hold up other wakes, is reported to the worker as the next deadline, and keeps the company selected after a restart; once accepted, it settles. - Thirty rejected escalations do not hide a later eligible one (paging). - A child with a cancelled blocker stays blocked with its board escalation. - A child whose last status decision came before the block moves; one decided after the block stays blocked. - A stranded child whose setup failed for another reason stays blocked with its board escalation. - A task bound to the workspace it created (the parent) is never moved. - Related suites pass: `subtask-workspace-independence`, `heartbeat-workspace-busy`, `heartbeat-remote-workspace-busy`, `heartbeat-held-comment-wake`, `recovery-deadline-acceptance`, `heartbeat-retry-scheduling`, `task-workspace-binding` (278 tests), and after the review changes `issue-blocker-stall-wakeups`, `issue-recovery-deadlines`, `issue-recovery-work-lifecycle`, `recovery-durable-wait-projection`, `owner-gone-attestation` (124 tests with the two workspace suites). - `pnpm verify`: exit 2 ("broader verification required", because `doc/execution-semantics.md` is a global CI input). Required CI supplies the broad coverage. - `npx tsc --noEmit -p server/tsconfig.json`: passes. ## Risks - The pass changes a Blocked task to `todo` only under narrow guards, and only for a task bound to a relative's workspace whose last run failed setup on that workspace's writer hold. A Blocked status from another decision, or an unresolved blocker, keeps the task blocked. - The moved child starts again from the project base. This is the product rule for subtasks. Nothing of its own work is dropped, because its runs failed before any provider work. - The pass runs inside the existing active recovery action sweep and selects at most 25 escalations and 25 pending wakes per pass. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, browser read-only API checks) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 12:15:25 -06:00
DottaandClaude Opus 5.5	34c1e6469c	fix(storage): retry small uploads and resume broken downloads (#20313 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A task's files live in a sandbox. Paperclip saves them to object storage when a run ends and restores them when the next run starts > - Object storage has short blips: a single 500, a reset connection, a download cut off halfway. Each of these used to fail the whole save or restore once, with no second try > - This pull request makes a short blip invisible. Longer outages are handled by the storage-unavailable wait (#20299 and a follow-up) > - The benefit is that tasks keep moving through ordinary storage failures without anyone clicking Retry ## Linked Issues or Issue Description Refs #19831. Storage chaos waves `w-20261109T145628-s9902` (ST1) and `w-20261109T151711-s9903` (ST2). - A single-request upload (files up to the multipart threshold, which includes most workspace archives) sent once. A 500 or connection reset failed the save. - A download whose body broke mid-stream failed the restore, even though S3 can resume from the byte offset. Related: #20299 (storage outage is shown as an infrastructure wait instead of a failure). ## What Changed - `server/src/storage/s3-multipart.ts`: `sendFileRequestWithRetry` resends a single file request and records throttles and backoff in the save timing. It uses the same throttle backoff as multipart parts for 429/503/SlowDown and up to the same part attempt count for other failures. It honours the caller's abort signal. - `server/src/storage/s3-provider.ts`: - A file-backed `putObject` under the multipart threshold uses that retry and reopens the source file for each attempt, so a resend never reuses a half-read stream. The Content-MD5 is computed once from the file and still verified by S3 on each attempt. - `getObject` resumes a body that fails with a transient socket error (reset, timeout, premature close and similar). It continues with a ranged GET from the byte where the stream stopped, at most three times. The resumed request carries `If-Match` with the first response's ETag, so a changed object fails instead of mixing bytes. Body-failure diagnostics are still recorded on every response. Destroying the returned stream closes the S3 body it is reading, even mid-read, so read timeouts and client disconnects still release the connection. - Tests: - `server/src/__tests__/storage-s3-transient-retry.test.ts` (new) runs against a local HTTP server speaking S3. ## Verification Red/green (each new test fails against master and passes with this PR): - `storage-s3-transient-retry.test.ts`: - A small upload is resent after a 500 and after a connection reset, re-reading the file. The second PUT carries the full content. - A download cut off mid-body resumes with `Range: bytes=<offset>-` and `If-Match`, and returns the complete bytes. - A download whose object changed before the resume fails with a precondition error and does not return mixed bytes. - A stalled download that its reader destroys closes its connection. Regression: the storage and workspace-finalization suites pass locally, and `tsc --noEmit` passes for `server`. ## Risks - Resending an upload is safe because each attempt writes the full object with the same Content-MD5. Resuming a download is bounded to three resumes and refuses a changed object. - No schema migration and no API change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 12:10:57 -06:00
Dotta	a3e7e63330	Guard Native recovery with process-space proof and explicit containment (#19813 ) ## Thinking Path > - Paperclip manages agents and their work. > - Recovery must preserve execution ownership and accepted results. > - A saved PID proves nothing outside its recorded process namespace. > - Uncertain unfinished work must stay held until its original owners stop. > - Operators need a source-bound containment action and a separate ordinary Retry. > - Accepted results must use the existing finalizer without another provider turn. > - This PR keeps those paths safe, recoverable, and quiet between real obligations. ## Linked Issues or Issue Description Refs: #19713 Refs: #19591 Recovery could use a local PID to release or replace a controller from another process namespace. Connected-chat Retry also rejected successful containment on a task in `todo`. The new containment action used a broader permission than the existing owner-stop action. A later review found that result-less guards also rejected the existing accepted-result confirmation action. ## What Changed - Record and verify the controller's kernel/PID namespace. Keep unknown and foreign unfinished ownership held. Preserve independent durable stop evidence and the existing remote lease fences. - Use the shared accepted-result ownership reconciler. Bind settlement to the observed lease token, generation, boot, PID, namespace and process birth. Retain finalizer, cleanup, current-controller, live-owner and remote exclusions. - Share the remote-owner reader across recovery consumers. Scope runner leases to the same company, task and run. Do not make local PID probes for remote accepted-result ownership. - Gate original-host containment on `runtime:confirm_owner_stop`. Bind it to the current task, source, assignee, original expiry, lease and Stop state. Audit separate controller/provider containment and reviewed action outcomes. - Keep containment separate from ordinary Retry. Use one canonical committed receipt and the normal admission, budget, workspace and lineage checks. Preserve the recorded external-action outcomes. - Keep the controller-only confirmation actionable for both result-less work and accepted results. Result-less work gets its next source-bound attention in the same transaction. Accepted results keep their original result and use normal shared settlement/finalization. - Keep identity inspection quiet. Preserve the original foreign-owner expiry and use the existing scoped finalizer in accepted-result recovery controls. - Update the recovery card, validators, execution projection, docs and access maps. Reuse existing canonical columns; add no migration or ownership backfill. ## Verification Current publication head: `71ea76b568de09d6313d8992e803131d9d5c54f3`. The current fix has a causal red/green check. On the test-only revision, both foreign-local stopped/fenced controller cases failed because recovery probed saved runner, process-group and provider PIDs in this host's namespace. This measured the wrong probes; it did not run duplicate providers. After the fix, the same two cases passed every no-probe, no-claim, no-controller-stamp, no-stop-receipt and unchanged-authority assertion. Three existing remote preparation, grace and fenced-controller progress cases also passed. Total: 5 passed, 0 failed;170 filtered rows excluded. All three normal affected App TypeScript programs passed on this head (660,681 and428 roots). The fix separates a controller wait from execution stop proof. Independent live/grace evidence still schedules recovery. A stopped/fenced foreign controller can pass this namespace guard only for remote execution, which keeps its later checkpoint and never-launched checks. Foreign local execution stays held before runner or provider PID inspection. Earlier source-scoped results remain evidence, not fresh executions on the current head: - On `c86dd741`, the complete failed-owner file passed 25/25, with no skips or filters. Empty scans wrote no controller stamps. Foreign ownership kept its original expiry. Normal returned-claim settlement and attention at the exact deadline passed. - On `c86dd741`, 15 selected accepted-result/restart controls passed. The81 filtered rows are excluded. Both completion cases used the existing scoped finalizer and kept their accepted result without another provider turn. - Trusted CI at synthetic merge `50d072a5` (actual master `1b7fe334` plus `c86dd741`) passed the complete 17-case maintenance-server file. This includes the original90-real-second quiet observation and two qualified hourly windows. The quiet case took118,918ms. The four jobs that failed on the earlier publication all passed on that synthetic merge. - On `6243b32c`, the complete ownership-report file passed 29/29. Native NULL namespace stayed unprovable and generic compatibility remained. Three selected recovery controls passed;170 filtered rows are excluded. - On `f81f2c3c`,356 business cases passed: whole ownership-route221, recoveryUI70, owner-proof38 and controller-confirmation8; selected accepted-result authority18; and one real connected-chat containment, separate authorized Retry, useful Native result and downstream reply case. The2,024 filtered rows are excluded. A separate earlier accepted-result causal case kept the original result and completed after restart without replay. Normal Shared exports, App programs, UI typecheck, token gates and map checks passed on their recorded revisions. Unchanged DB, adapter, Eval and Runner inputs/outputs have explicit carry proofs. The clean normal merge with fetched master `8fe524f7` preserves all31 exclusive owned paths and14,827 unrelated master entries. The nine overlapping paths were reviewed, including the upstream unidentified-run stop action. Fresh CI must validate that actual merge composition. PG cases used existing external PG17, fresh owned databases, pools at most4, one worker and no SQL observer during cases. Registered clients and recorded process groups closed. Independent final SQL audits found zero campaign sessions before normal database cleanup. Original failing fixtures remain retained. Original failures are preserved. One new test first failed compilation due to a missing getter import; an import-only correction passed its full owning programs before the causal red/green check. A local quiet attempt stopped in the normal prerequisite hook before any business case and its collector rejected two normal build markers. The later17-case CI pass establishes quiet recovery; it does not turn that failed local collector into a pass. An earlier useful case passed before its private cleanup collector failed on an owned read-only directory; normal cleanup completed after all owners closed. Earlier runner-shutdown cancellation and a separate private whole-suite failure remain unexplained. Current-head checks are complete:57 success,1 neutral and3 skipped, including the successful aggregate CI gate. The legacy status passed. Greptile is5/5 and Ponytail gives an explicit5/5 Ship verdict for this head. All seven review threads are resolved. Full recursive typecheck, the full test suite and release build were not repeated locally. No result here proves deployed suspension, a live provider or termination on another host. ## Risks - Unknown physical ownership can require an owner/admin action. A false containment statement can be unsafe. - New work, reassignment, Stop, renewed authority, accepted results or pending cleanup can invalidate an action. The operator must refresh. - Controller confirmation does not prove provider/tool containment. Completed external actions must not be replayed. - Generic legacy and child-process compatibility remains separate from the strict Native controller policy. - Existing saved state and canonical schema are preserved. Permanent safety refusals remain held with their normal named recovery path. ## Model Used OpenAI `gpt-6.1-sol` agents implemented and tested the changes with reasoning, repository tools and code execution. A GPT-6-based Codex managing agent reviewed and integrated them. The host does not expose the manager's exact build or the context-window sizes. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs: #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge	2026-11-09 12:06:11 -06:00
DottaandClaude Opus 5.5	365c771030	fix(chat): give project and workspace tool results their page links (#20324 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - People ask a chat agent to take them to a project or to the files of a task > - The agent finds the right project or workspace, but it cannot link the page > - The `list_projects` and `list_workspaces` tool results have no page route, and the chat link guidance names no project or workspace format > - The agent then writes a bare path or a shortened directory that the person cannot open > - This pull request adds the page `href` to these tool results and adds the project link format to the chat guidance > - The benefit is that chat replies give a clickable link to the real project or workspace page ## Linked Issues or Issue Description What happened In chat, a person asked "Take me to the Launch project and show its open tasks." The agent found the project. Its reasoning said "There's no project page route documented explicitly". The reply wrote the project as a bare path `/COM/projects/<id>`, which is not clickable. In a second chat, a person asked where the files of a task are. The agent named the correct workspace but gave a shortened directory and no link. Expected behavior The reply links the project page (`/<prefix>/projects/<id>`) and the task's workspace page (`/<prefix>/execution-workspaces/<id>`) as Markdown links, so the person can open them. Steps to reproduce 1. Create a company with a project and a task that has a saved execution workspace. Enable isolated workspaces. 2. In chat, ask the agent to take you to the project, or to say where the task's files are. 3. Look at the reply. It has no project or workspace link. Related: Refs #17985 (an earlier open PR that adds project and goal link wording for newly created projects only). This PR covers project discovery and workspaces and adds tool-result hrefs. ## What Changed - `list_projects` rows now carry `href: /<prefix>/projects/<id>`. Both the native runner tool authority and the MCP project-tools route pass the company prefix. - `get_workspace` and `list_workspaces` now carry `href: /<prefix>/execution-workspaces/<id>`, only while isolated workspaces are enabled. The workspace route redirects to the dashboard when the feature is off, so no href is offered then. - The chat link guidance (`AGENT_CHAT_DIRECTIVE`) adds the project link format. It tells the agent to link a returned `href`, and to name a workspace that has no `href` instead of linking it. - Regression tests: guidance text against the registered UI route, `callProjectTool` href, and an embedded-Postgres authority test for project hrefs and the workspace flag on/off. ## Verification - Red then green: the three new tests fail on `origin/master` source and pass with this change. - `npx vitest run` in `server/` for: `agent-conversations`, `task-workspace-api`, `project-tools-pagination`, `task-workspace-tools`, `chat-project-tools`, `dot-runner`, `heartbeat-list` (prompt-size limits), `task-markdown`, `muse-native-visibility`, `paperclip-runner-tool-authority`, `runner-api`, `codex-local-execute`, `board-access-denial`. All pass. - `npx tsc --noEmit -p server` passes. - Live chat eval: see the comment on this PR for the paid runner E2E run of `navigation-preferences.find-project-view` and `workspaces.locate-files` with this change. ## Risks - Low risk. The change adds one optional field to read-only tool results. The output schemas allow extra fields. - The guidance grows by two sentences. The prompt-size test in `heartbeat-list.test.ts` passes. - `list_projects` and the workspace tools now do one extra small read (company prefix, and the instance experimental settings for workspaces). ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), agentic coding with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 11:52:44 -06:00
Devin FoleyandPaperclip	0f84df91da	fix(native): accept a completed child's file in the parent's completion report (#20325 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - In onboarding, the agent plans the user's first task, creates one child task for the work, and waits for it > - The `first-task` skill tells the agent to publish a requested file on the child task. When the parent wakes, the agent links the child's file and does not publish a copy > - Native `register_deliverable` is bound to its current task, so the parent cannot register the child's file again > - The native completion check required each cited attachment to be on the task being completed > - So when the parent cited the child's download link, `paperclip_finish` failed, and the agent reported the finished onboarding task as blocked > - This pull request lets a completion report cite an attachment on a direct child task that is done > - The benefit is that onboarding closes after its child task delivers a file, and the user does not see a false "blocked" status ## Linked Issues or Issue Description Related: #18258 (onboarding artifact completion instructions). What happened? An onboarding run created a child task to build and publish a project. The child task registered `project.zip` with `register_deliverable` and was marked done. The parent woke with reason `issue_children_completed`. The agent called `paperclip_finish` with `reportedWorkDisposition: "done"` and cited the child's `/api/attachments/<id>/content` link. The call failed two times with: ``` Completion cites no registered attachment on this task. Use register_deliverable for the requested file and cite deliverable:<attachmentId> from its receipt. ``` The skill does not let the agent register the file on the parent, so the agent called `paperclip_block` with "Paperclip completion validation" as the owner. The parent task stayed `blocked`. Nothing wakes it again. Expected behavior The parent completes and links the child's file. Steps to reproduce 1. Create a task with a child task. 2. In a native run on the child, register a file with `register_deliverable`, and mark the child done. 3. In a native run on the parent, call `paperclip_finish` with `reportedWorkDisposition: "done"` and the child's `deliverable:<attachmentId>` or `/api/attachments/<attachmentId>/content` as evidence. 4. The call fails with "Completion cites no registered attachment on this task". ## What Changed - `server/src/services/native-runtime/native-deliverable-feedback.ts`: when a cited attachment is not on the task, `validateNativeDeliverableEvidence` also looks for it on a direct child task with status `done`. If it is there, the citation is accepted. - The child's file is context, the same as an attachment from an earlier run on the same task. It does not count as this run's publication of a file that the parent's own request names. - With `lockSources`, the finalization path locks the attachment and child task rows with `FOR SHARE NOWAIT`, as it does for other sources. - The error text now says "on this task or its completed child tasks". - This applies to the `paperclip_finish` check and to the two finalization checks, because they use the same function. - `doc/AGENT-ARTIFACTS.md`: the native runner section states that a completion report can cite a file on a completed direct child task. ## Verification - Extended "prepares one verified same-run attachment and replays without duplicates" in `server/src/services/native-runtime/native-runner-file-handoff.test.ts`: - An attachment on an unrelated task is still rejected. - An attachment on a child task that is not done is still rejected. - An attachment on a done child task is accepted, through `paperclip_finish` and through the locked finalization path. - When the parent's own request names a file, the child's file alone does not satisfy it. - Without the source change, the test fails with the same error as the run above. - `npx vitest run` on `chat-attachment-reuse`, `native-completion-feedback`, `native-completion-format-feedback`, `native-deliverable-feedback`, and `native-runner-file-handoff` tests in `server/src/services/native-runtime`: 281 passed. - `npx vitest run src/__tests__/native-result-commit-work.test.ts src/__tests__/native-status-arbiter-corpus.test.ts` in `server`: 167 passed. - `npx tsc --noEmit -p server/tsconfig.json` passes. ## Risks - Only direct children with status `done` are accepted. A grandchild's file, or a file on a child that is not done, is still rejected. - A parent that names a requested file in its own request still needs its own delivery evidence. This keeps the current rule for earlier outputs. - A task that is already blocked by this error does not change. The user can mark it done, or comment so the agent runs again. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (code reading, database queries, shell commands, and test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-09 09:44:17 -08:00
DottaandClaude Opus 5.5	47dccbf3eb	fix(recovery): re-prove an exhausted save's agent stop and free the workspace (#20295 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When an agent run ends, Paperclip saves the files that the run changed in its remote workspace. Until that save settles, the run holds the workspace, so no other task can write it > - When the save fails, a bounded system recovery proves that the agent stopped (a provider stop receipt) and then saves the files again > - If the stop cannot be proven before the 30-minute recovery window ends, the recovery is exhausted with `agent_stop_unconfirmed`. Exhaustion was terminal: Paperclip never proved the stop again, even after the run ended and its sandbox had a stop receipt > - The hold then stayed forever. On staging, a child task's save was exhausted this way on Nov 5. Its parent and its two sibling tasks share that workspace, and all of them stayed blocked for four days > - This pull request re-arms such an exhausted save automatically, on a bounded backoff, once the run is provably over. After the bounded re-arms, the hold continues from the last save with the unsaved files attached > - The benefit is that a dirty workspace hold always settles without a person, and no unsaved work is discarded silently ## Linked Issues or Issue Description Refs #20273 (the stall sweep stops failing wakes against this kind of hold) Refs #20278 (a settled restore hold returns its warm sandbox) Refs #19860 (subtasks get their own workspace) What happened? A child task's run wrote its remote workspace and failed to save it (`native_workspace_sync_out_retry_exhausted`). The save recovery made one attempt. That attempt failed with `agent_stop_unconfirmed`, and the recovery window ended before a second attempt. The recovery became `exhausted`. Ten minutes after the first attempt, the sandbox lease got a provider stop receipt, so the agent was provably gone. But nothing looked at the exhausted recovery again. The run kept the workspace's writer hold. The parent task and every other task on that workspace could not start ("The workspace still has an unfinished writer or save operation."). The blocker told a person to press Retry. Expected behavior An unproven stop is a wait, not a refusal of the saved files. When the run is over, recovery must prove the stop again and finish the save, so the hold settles automatically. If the stop still cannot be proven, Paperclip must keep the unsaved files and then release the hold. A person must not need to act. Steps to reproduce 1. Run a task in a remote (sandbox) workspace. Make its final save fail. 2. Keep the workspace recovery from proving the agent stop until its 30-minute window ends (for example, keep the finalization ownership busy). 3. See that the recovery action is `exhausted` with `agent_stop_unconfirmed`, and that it stays so after the lease gets a stop receipt. 4. Wake another task bound to the same workspace. See that its setup fails because the workspace is held. Root cause - `server/src/services/workspace-recovery.ts` (`recoverRetainedWorkspace`): an `exhausted` state only goes to the sibling settlement and to the retained-source fallback. The fallback accepts only permanent source refusals (`isUnverifiableSourceFailure`). Every other exhausted state is handed to the board and never attempted again. - `recheckStoppedWorkspaceRecovery` skips `exhausted` states, and it only accepts acknowledged cancellations. A failed run whose lease got a stop receipt later is never re-examined. - `retainedWorkspaceCandidates(…, obligationsOnly)` and `nextWorkspaceRecoveryAt` do not select exhausted `agent_stop_unconfirmed` holds, so the recovery worker never wakes for them. ## What Changed - New `server/src/services/workspace-recovery-stop-proof.ts` (`rearmExhaustedStopProof`). For an exhausted save whose last failure is `agent_stop_unconfirmed` (`recheck_evidence`), it starts a new bounded system recovery generation. It does this only when the source run is provably over: the run is terminal, no local execution remains, and no controller lease or finalizer lease is live. The new generation repeats the strict stop proof (a provider stop receipt) and the save. It records an `issue.workspace_recovery_rearmed` system activity. - Re-arms are bounded: at most 4. The first is due at once. The next ones are due 30 minutes, 2 hours and 8 hours after the previous re-arm. The `stopProofRecovery` marker in the recovery action evidence records the count and the next due time. - `workspace-recovery-fallback-discovery.ts`: new discovery conditions (`stopProofRearmPendingCondition`, `stopProofRearmDueAt`, `stopProofRearmDueCondition`) and `retainedHoldFallbackEligible`. `retainedWorkspaceCandidates` (worker pass) and `nextWorkspaceRecoveryAt` use them, so the recovery worker wakes when a re-arm is due. - `workspace-recovery-fallback.ts`: when every re-arm is spent and the stop is still unproven, the hold takes the existing retained-source fallback. The unsaved files are attached to the task (unapplied, secrets omitted), the unverified epoch is abandoned, and the task continues from the last save. - `system-recovery-state.ts`: the exhausted `agent_stop_unconfirmed` text now says that Paperclip tries again automatically. It no longer asks a person to press Retry. Retry stays available. - Review follow-ups: the re-arm applies to every workspace recovery the worker owns (restore and historical holds too). A due re-arm is scheduled no earlier than the source run's controller or finalizer lease expiry, so the worker is not woken every 15 seconds while a lease is live. A fallback whose bounded checks are spent is reconsidered once when an allocation of the source run changes, and `settleStopOnlyCleanup` wakes the workspace recovery worker when it records a stop receipt. The automatic-retry text is shown only while a re-arm remains. - `doc/execution-semantics.md`: documents the re-arm, its bounds, the fallback, and the upgrade behavior. ### Previous-release story (`rules/upgrade-safe-run-state.mdc`) - Carried forward, no migration. A hold exhausted by an older release has no `stopProofRecovery` marker. The recovery worker treats it as due at once and re-arms it on its first pass after deploy. The new regression test saves the exact Nov 5 evidence shape (`attempts: 1`, `retriesUsed: 0`, `state: exhausted`, no marker) and resumes it with the new code. - Mid-run deploy. A run that is still running, or whose controller lease or finalizer lease is live, is not re-armed. The test covers a live controller lease. The re-arm uses the same CAS (generation and revision) and the same row locks as the other recovery writers, so an old worker's late write cannot overwrite the new generation. - No new refusal codes. No saved run identity changes. The re-arm writes only recovery-action evidence. ## Verification - Red (before the fix), `npx vitest run src/services/workspace-recovery-stop-proof.test.ts`: 3 failed, 1 passed. The staging-shape test failed because the worker did not wake (`nextWorkspaceRecoveryAt` returned `null`) and the hold stayed. The backoff and fallback tests failed for the same discovery gap. - Green (after the fix): 4 of 4 pass. - Staging shape: a parent and 3 children that share one remote workspace. One child's save is exhausted with `agent_stop_unconfirmed`, and its run ended days ago with a stop receipt. Before recovery, the hold refuses the parent (`remoteWorkspaceHoldRefusesStart` is `true`). The recovery worker discovers the hold, re-arms it, proves the stop, saves the files and completes. The workspace is not dirty, it has a new snapshot, and the hold is `free` for the parent. The action stays system-owned and its text asks no person to act. - A live controller lease keeps the hold. No re-arm happens and no lease is resumed. - A re-arm that is not yet due is not discovered. The worker's next wake is in the future, and the text says that Paperclip tries again automatically. - After 4 re-arms, the hold continues from the last save. The unsaved files are captured as a task attachment, and the hold is `free`. - A live controller lease defers the worker's next wake to the lease expiry, and the hold is not selected before it. - A spent fallback is not selected, and a later lease change (a late stop receipt) makes it selected again; the worker then settles it. - `system-recovery-state.test.ts`: the automatic-retry text appears only while a re-arm remains. - Related suites, all pass: `src/services/workspace-recovery`, `src/services/remote-workspace`, `src/services/system-recovery-state.test.ts`, `src/services/task-retry.test.ts`, `src/__tests__/heartbeat-observed-workspace-preservation.test.ts` (34 files, 1510 tests), plus the stop-cleanup suites (`environment-retained-orphan-sandbox`, `live-recovery-work`, `retained-native-session-cleanup-work`, `legacy-workspace-restore-recovery`, native workspace warm-reuse, cancelled preservation, reconstruction and failed recovery; 11 files, 622 tests). - `pnpm verify`: exit 2 ("broader verification required", because `doc/execution-semantics.md` is a global CI input). The focused suites above cover the change, and required CI supplies the broad coverage. - `npx tsc --noEmit -p server/tsconfig.json`: passes. ## Risks - A re-armed generation resumes the stopped sandbox to save its files. This is the same physical path as a manual Retry. It runs only when the source run is provably over. - Re-arms are bounded (4) and spaced, so a stop that can never be proven does not loop. After that, the existing retained-source fallback settles the hold. If the retained files cannot be read, the fallback records why. This is the same behavior as for a refused source. - The parent's workspace receives the child's saved files, because the child (bound by an older release) wrote that workspace. This is the existing #19860 behavior: the files that a legacy child left in the shared workspace stay there. #20314 moves the stranded legacy children to their own workspace. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, browser read-only API checks) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 11:44:15 -06:00
DottaandClaude Opus 5.5	8fe524f796	fix(issues): let the board confirm an unidentified run stopped instead of an endless Retry wait (#20310 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Before a new attempt can start on a task, Paperclip must prove the previous run's agent stopped, so two agents never work the same task at once > - A local native run can be interrupted after it asked to start the runner but before it recorded the runner's process. The runner may have saved state and started a provider, and provider processes can outlive it, so the existing check (#19694) rightly refuses to assume the run stopped > - Today that case has no way out: every Retry is accepted and then waits forever as `process_identity_missing` (storage chaos ST1, t02/t03) > - This pull request makes the wait explicit and actionable. The task explains why it waits, and a board operator who has checked can confirm the run stopped; the parked Retry then starts at the click > - A second shape from the Daytona storage baseline (t06) parked a Retry with no reason at all: a follow-up was interrupted by a graceful shutdown while it was still restoring its workspace. No provider ran, but the restore hold kept the same agent held, so the Retry never started > - The benefit is that no Retry stays accepted-and-waiting with nothing anyone can do ## Linked Issues or Issue Description Refs #19831. Storage chaos wave `w-20261109T145628-s9902` (ST1), tasks t02 and t03. What happened (CHA-3, run `3c3f6241`): - The run requested a runner launch (`native.process_start_requested`), and a K2 restart hit before its process identity was recorded. - After the restart, the resumed attempt failed on the storage outage. The task was blocked. - The board's Retry (202) was parked as `executionWait.reason = process_identity_missing`: "The previous run has no verified stop record." - `recordUnidentifiedLocalRunnerStop` cannot prove the stop because this run's runner saved control-plane state. Once that exists, a provider may have started (#19694), and a provider is not guaranteed to carry `--run-id` or `PAPERCLIP_RUN_ID` (the Codex provider env omits it on purpose). The shared owner prover needs a PID, which the run never recorded. - Nothing could ever clear the wait. Daytona storage baseline `w-20261109T154810-s9911`, task t06: - The run succeeded. The follow-up was cancelled (`continuation_task_ownership_changed`), and the next run was interrupted by a graceful shutdown (`server_shutdown_interrupted`) 3 s into `preparing`. It was restoring the saved remote workspace and had launched no provider. - Restore recovery saved the files (`workspaceRecovery.status: recovered`, system recovery `complete`). But the `legacy_execution_requires_reconciliation` restore hold releases only for a different owner, or for a native Stop fenced before any provider turn, so it kept the same agent held. - The UI Retry returned 202 and parked with `executionWait.reason: execution_recovery`, the generic wait, so the task showed a Retry that never started. Automatic proof needs the runner to record its identity before saving state and to group provider processes under it. That is follow-up runner work; this PR removes the endless wait now. ## What Changed - `server/src/services/native-local-process-stop.ts`: new `confirmUnidentifiedLocalRunnerStopped`. It accepts only a finished native run of the task with no recorded process and whose latest launch evidence is an unconfirmed launch on this host in a local environment. It still refuses while a runner with `--run-id <run>` is running. It records `native.local_process_stopped` with `basis: "operator_attested"` and the attesting user. That is the receipt the existing continuation gate already accepts. - `server/src/routes/issues.ts`: `POST /issues/:id/recovery-actions/confirm-run-stopped` (`runId`, `stopEvidence` of 20 to 12,000 characters). It is board-only and requires issue read plus `runtime:confirm_owner_stop`, like the other stop attestations. It logs `issue.run_stop_confirmed` with the evidence, promotes deferred intents for the issue, and resumes each parked task Retry, so work starts at the click. Refusals return 409 with a code. - `packages/shared`: `confirmRunStoppedSchema`, and the presentation user action `confirm_run_stopped`. - `server/src/services/task-interruption-presentation.ts`: a Retry parked as `process_identity_missing` for a native run that recorded no process now presents as waiting for the user (`canRetry: false`, `userAction: "confirm_run_stopped"`), with a message explaining why and who can act. A run with a recorded process keeps the existing verification wait. - UI: - `ExecutionBlockerNotice` shows the explanation and, for users who can confirm a stop, "Confirm it stopped" with a short evidence form. - It reuses `PreviousServerStopConfirmation`, now with optional `action`, explicit `runId` and `formId`, and a custom button label. - The Inbox names the action. - `server/src/services/execution-blocker.ts`: `interruptedPreparationBeforeProviderWorkCondition`. The restore hold also releases for the same agent when the stopped run was a system-interrupted preparation (legacy runtime never resolved, still `preparing`) with no process identity, no runner or provider events, no finalization or result, and every lease released with a confirmed stop. It mirrors the existing interrupted-preparation proof and still requires the completed workspace-recovery receipt. Auto-replay is unchanged; this only lets the user's Retry or message start. - Review fixes: - `unidentifiedLocalLaunchRefusal` holds the endpoint's launch, host and lease checks, and the presentation offers `confirm_run_stopped` only when it passes. - The attested stop wakes cost accounting in the same transaction. - The confirmation form is keyed by task and run. - The route is registered in OpenAPI. ## Verification Red/green (each new test fails against master and passes with this PR): - `server/src/__tests__/issue-confirm-run-stopped-routes.test.ts`: - The parked Retry presents `confirm_run_stopped` with the explanation. - Confirming records the operator-attested stop and the activity entry, satisfies the stop check, and resumes the parked Retry: its receipt is coalesced into a newly started run. - It refuses with 409 while a runner for the run is running. - Agents and viewers get 403. - `ui/src/components/ExecutionBlockerNotice.confirm-run-stopped.test.tsx`: the notice offers "Confirm it stopped" (and no dead Retry) and posts the run and evidence. A viewer sees the explanation without the action. - `server/src/__tests__/interrupted-preparation-restore-retry.test.ts` (t06 shape): - The restore hold no longer blocks. - A Retry is accepted and resumed into a started run, and the audit hold keeps `replay: blocked`. - The hold stays when a provider turn started, when a runner event was recorded, when an operator stopped the run, or when the sandbox has no stop receipt. - `issue-confirm-run-stopped-routes.test.ts`: - A case per refusal code checks the presentation keeps the generic wait and the endpoint returns 409. - An `alreadyStopped` case. - An accounting-wake assertion. - The UI test switches targets with stale evidence. Regression and static checks: - `ExecutionBlockerNotice`, `WorkspaceExportRecovery` and its control tests: 74 passed. - `tsc --noEmit` passes for `packages/shared`, `server` and `ui/src`. `pnpm check:token-gates` is clean. ## Risks - A board operator can now attest that an unidentified run stopped. The attestation is permission-gated, requires written evidence, is activity-logged, and is refused while a runner for the run is visibly running. A wrong attestation could start a second agent beside an orphaned provider, the same trust model as the existing previous-server attestations. - The confirmation component now accepts a run without a recovery action. Existing callers pass the action as before. - No schema migration. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 11:36:41 -06:00
DottaandPaperclip	ff2d463eed	feat(cli): manage agent grants and keep remote login on the public origin (#20323 ) ## Thinking Path > - Paperclip lets people control their AI agents and permissions. > - Remote operators need a browser-approved CLI login. > - Proxies can send that login's approval link to an internal host. > - Human member commands cannot grant an agent an explicit permission such as commentary:view. > - This change keeps approval on the canonical public origin and adds Board-only single-grant commands. > - The API checks company access and permission-management authority and records each change. > - An operator can sign in, grant one agent permission, and preserve the agent's other grants. ## Linked Issues or Issue Description What happened? A proxied CLI login returned an approval URL on the upstream host. The signed-in browser was on the public host. The CLI also had no command to add an arbitrary explicit agent grant: member commands manage human members, while agent permissions:update exposes fixed configuration fields. Expected behavior Approval uses the configured canonical public origin. A Board operator with users:manage_permissions can grant or revoke one explicit agent permission without replacing other grants or activating an inactive membership. Steps to reproduce Configure a public auth origin. Create a CLI challenge through a proxy that replaces forwarded-host headers. Inspect the approval URL. Then try to grant commentary:view to an agent through the CLI. ## What Changed - Use the existing public-origin resolver for CLI approval URLs. - Add agent permission:grant and permission:revoke commands. - Add PATCH /api/agents/{id}/permission-grants with strict shared validation and Board-only OpenAPI metadata. - Serialize each mutation on the agent row. Preserve unrelated grants and membership state. Commit the grant and its activity record together and publish activity after commit. - Document remote login and single-grant management. ## Verification - The new approval-link regression failed before the fix and passes after it. - All 25 CLI auth route tests pass. - All 72 affected CLI, login and OpenAPI tests pass. - All 5 PostgreSQL grant tests pass, including actual commentary reads (403 before grant, 200 after grant, 403 after revoke), preservation, audit attribution, concurrent additions, invalid inputs, inactive or missing membership, agent self-grants, cross-company denial, and rollback on audit failure. - Shared and CLI typechecks and server declaration-layer checks pass. - pnpm verify requests broad CI coverage because doc/CLI.md is a global verification input. Greptile reviewed final head `338c58fb24` at 5/5 with no findings. All 58 non-skipped checks pass on that head. CI run: https://github.com/paperclipai/paperclip/actions/runs/42663640985. Live target acceptance is in progress. ## Risks This adds an explicit permission-management API. It requires Board authentication, company access, and users:manage_permissions. Foreign-company agents return 404. Granting a key intentionally makes that key unscoped in its company; revocation removes that key. Other keys and agent configuration remain unchanged. Inactive memberships and unapproved or terminated agents cannot gain grants. No schema change is needed. Revert this change to remove the new management surface; existing grants retain their normal authorization rules. ## Model Used OpenAI GPT-6 through Codex, with reasoning, code editing, shell tools, and browser testing. The runtime does not expose a more specific model ID or context window. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-09 11:35:52 -06:00
Dotta	1b7fe33432	fix(workspaces): preserve import verification across process restarts (#20311 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Online import preserves old local files before the web server can use a disposable filesystem. > - A parent import must authenticate saved child archives before it can omit their directories. > - The operation journal preserves import identity, but completed child checks existed only in process memory. > - A container restart therefore repeated downloads, decryption and restoration that had already succeeded. > - This pull request saves a versioned verification receipt on each immutable child snapshot. > - A replacement process can reuse those checks while it reacquires locks and verifies current local files. ## Linked Issues or Issue Description Refs #19854 for verification reuse within a live source fence. Refs #19873 for cohort import. Refs #20298 for the separate remote-hold admission fix. Searched related workspace verification and restart PRs; none provides durable child verification. What happened? Replacing a server container interrupts a long online import. The original operation and snapshot survive, but a later invocation downloads, decrypts and restores completed child archives again. The current cache ends with the source-lock owner. Expected behavior Completed immutable checks survive process and scratch loss. Resume keeps the same operation and snapshot, reuses only correctly bound verification, and publishes once. Current local files and Git state must still be checked under fresh source locks. Steps to reproduce 1. Import two nested child workspaces, then start the parent import. 2. Kill the importer after a child verification finishes. Remove its scratch directory. 3. Resume in a new process. Before this fix, the completed child is downloaded and restored again. Paperclip version or commit The regression fails on the previous migration verifier. The final branch includes master at `189fefb99183953be37f0d1084275fbb0f35d6e2`. Deployment mode Managed Linux server with PostgreSQL and explicit S3 storage configuration. ## What Changed - Add optional opaque storage namespace identity. S3 binds it to the explicit endpoint, region, bucket, normalized prefix and addressing style. Implicit endpoints retain full verification. - Save a versioned child-snapshot receipt only after canonical crypto, manifest and file restoration succeed and a second HEAD confirms the same strong ETag and size. - Bind reuse to the complete snapshot reference, key material, child lineage, source operation and storage namespace. Require a fresh strong HEAD and canonical validation that the current secret provider can unwrap the saved snapshot key. - Keep current directory/Git capture and source-owner checks on every invocation. Read-only verifier callers do not write checkpoints. - Merge checkpoint metadata under a short row lock without changing parent journal hashes, snapshot retention or update timestamps. Bound reads and writes. Lock timeouts, statement timeouts, deadlocks and serialization conflicts are safe cache misses. Other errors remain fatal. - Extend real PostgreSQL/Git/archive tests with process death, scratch removal, resume, changed bindings and single publication. ## Verification - Baseline RED: two import invocations read each child twice. With durable receipts, each child is read once across both invocations. - Real process regression passed on the final rebased head: SIGKILL the owned importer after one checkpoint, delete all scratch, and resume in a new Node process. It retains the same operation and snapshot, reads each child once across both processes, and records one publication. - 53 focused cases passed in 104.86 seconds on the final production bytes before rebase. They cover existing nested/cohort behavior, namespace identity, changed source/reference/key/lineage/ETag, unknown proof versions, weak validators and bounded row contention. - After clean rebase and stronger scratch/snapshot assertions, the real process case passed again in 18.54 seconds. Server TypeScript, module boundaries and diff checks passed on final head `9da68ee7ce4a778201ef85e21a11c49eeea7bc16`. Required CI supplies final-head broad coverage. - The initial restricted-network dependency setup failed; unchanged preflight passed with normal network access. Initial TypeScript failures in test pipe assertions were corrected. No failed attempt is counted as a pass. - Review found a missing current-master-key check. A real restart regression with all child checkpoints complete reproduced unsafe publication after switching to a different valid master key. Follow-up `7ea92c2abd9ffdb2e7dd81cecf62c8fc7a6e74a5` delegates to the canonical snapshot-key owner before both durable and in-memory cache hits. - The fixed process regressions passed 2/2. Ten key-authority and existing identity/source guard cases passed. The wrong-key process now refuses with zero additional child payload GETs, no publication, and unchanged original journal/snapshot rows. Server typecheck, module boundaries and diff checks passed on the follow-up. - Review also found that transient database failures could make the optional checkpoint cache stop a valid import. Read and save now fall back only for SQLSTATE `55P03`, `57014`, `40P01`, and `40001`. Connection loss, identity changes, crypto failures, programming errors and cancellation remain fatal. Reads use a read-only transaction with one-second lock and eight-second statement limits. - The checkpoint database-fault regression failed two cases before the fix. All four recoverable import cases then passed. The final selected run passed 21 cases, including fatal-error and cancellation controls, real row contention, both process-restart cases and key/identity guards. Server typecheck and boundaries passed. Fault injection is limited to the checkpoint database boundary; the importer, PostgreSQL, Git, archives, storage and publication remain real. - Latest master was merged without rewriting published history. Six real import/process-restart cases passed again on final head `8c8e4213a680ebb5c8d6864416bc3a4f271e1e76` in 38.03 seconds. [Required CI](https://github.com/paperclipai/paperclip/actions/runs/42658248222) finished with 54 successful checks and four intentional skips; Snyk passed. [Fresh review](https://github.com/paperclipai/paperclip/pull/20311#issuecomment-6745354381) is 5/5 with all four threads addressed and resolved. Hosted verification after normal release delivery remains pending. ## Risks - A stale receipt could skip authentication of a different object. Complete reference/lineage/namespace binding, fresh strong HEAD and row-locked save validation prevent that reuse. Missing or unsupported hints trigger full verification. - Mutable local files can change between attempts. Their source locks and fresh captures remain required; a checkpoint never certifies current local bytes. - Checkpoint persistence can contend with other snapshot owners. It has one-second lock and eight-second statement limits; the four classified transient SQL errors retain the normal fully verified path. Other database errors stop the import so lost source ownership cannot be hidden. - No schema, archive format, public API or saved-run identity change. Older snapshots without receipts remain readable and get full verification. Existing data and recovery obligations stay intact. ## Model Used OpenAI Codex, GPT-6.1 Sol (`gpt-6.1-sol`), with reasoning, repository tools and code execution. Context-window size is not exposed by the runtime. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge	2026-11-09 11:03:48 -06:00
DottaandClaude Opus 5.5	0cf70c9d0c	fix(chat): tell overviews to state when the picture is from (#20301 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - People ask the CEO chat agent for a quick picture of what the company is doing now > - Such a picture is only useful when the person knows how current it is > - The chat guidance asks for accurate status, but it does not ask for a time > - A live overview said "right now" and gave no time, so the person could not tell when it was observed > - This pull request adds one guidance sentence: state the picture's time as a UTC clock time, from the latest message timestamp > - The benefit is that overviews tell the person when their information is from ## Linked Issues or Issue Description Refs #19614 What happened A person asked the chat agent "Give me a quick picture of what the company is doing right now." The agent read agents, projects, tasks and approvals and gave a correct summary. The summary did not say when it was observed. The only time word was "right now". Expected behavior An overview of current work states when the picture is from, as a clock time. The native prompt already lists each new comment with its timestamp, so the agent has a time it can use. Steps to reproduce 1. Open a chat with the CEO agent in a company that has active, blocked and done tasks. 2. Ask "Give me a quick picture of what the company is doing right now." 3. The reply summarizes the work but gives no observation time. ## What Changed - `AGENT_CHAT_DIRECTIVE` status-update guidance: when the agent describes what the company, a project or agents are doing now, it states a UTC clock time, such as "As of 15:26 UTC". It uses the latest message's timestamp unless a record it read gives a later time. - A regression test checks that the directive contains this guidance. ## Verification - `cd server && npx vitest run src/__tests__/agent-conversations.test.ts src/__tests__/heartbeat-list.test.ts` passes (includes the prompt-size limits). - The chat-catalog eval `activity.company-overview` now grades a real observation time (stacked eval PR on `chat-eval/master-candidate`). This fix is cherry-picked onto that eval branch for the next live run. No live run has used it yet. ## Risks - Low risk. This is one guidance sentence. Overviews get one extra clause with a time. - The time is the message time, not a server clock read. Reads happen seconds after the message, so the difference is small. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 10:56:38 -06:00
DottaandClaude Opus 5.5	12523fb677	fix(chat): send a task whose automatic recovery stopped to the person's Retry (#20306 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A person asks the chat agent (the controller) to act on tasks that other agents own > - When a task's automatic recovery stops, for example after three runner losses, the task is blocked and the person gets a Retry control on it. Retry rechecks the task's gates and keeps the failed attempt's evidence > - The blocker told every reader "Try again or send a new message to continue". A controller read that as its own instruction. It found that Retry is board-only, then resumed the task with a comment and reassigned the task to itself > - That skips the assignee's budget and hold gates and hides the failed-run Retry from the person > - This pull request names Retry as the person's control in the blocker text, and tells the controller to point the person to it > - The benefit is that "try that task again" leads to the supported, gated Retry instead of an agent takeover ## Linked Issues or Issue Description Refs #20270 (automatic recovery after a lost local runner) What happened? In the chat eval `chat-features.runner-opencode.local.recovery.retry-exhausted`, a person asked the CEO agent to "Try the website task again" after the website task's native recovery was exhausted. The agent read the task's `executionBlocker` (`canRetry: true`, nextAction "... Try again or send a new message to continue once the previous execution has stopped."). It found that `POST /api/issues/{id}/retry` is board-only. It then posted a `resume: true` comment, which moved the task from blocked to todo, and reassigned the task to itself to re-run it. The task lost its failed-run Retry control, and the original assignee's gates were not checked. Expected behavior The agent links the original task, says why it stopped, and tells the person to use Retry there. It takes the work over only when the person explicitly asks for that. Steps to reproduce 1. Let a native run exhaust its automatic recovery (for example, three runner losses). 2. In the controller chat, ask it to try that task again. 3. The agent comments to resume the task, or reassigns it, instead of pointing to Retry. ## What Changed - `execution-blocker.ts`: the text that is added when a person can retry is now `EXECUTION_BLOCKER_PERSON_RETRY_ACTION`: "A person can use Retry on this task, or send it a new message, to continue once the previous execution has stopped. Agents cannot retry it for them." - `agent-conversations.ts`: a new chat guidance paragraph. When another task's automatic recovery has stopped, the supported next step is the person's Retry on that task. The controller links the task and explains why it stopped. It does not reassign the task, change its status, comment on it to resume it, or create a replacement task, unless the person explicitly asks it to take the work over. ## Verification - `board-access-denial.test.ts`: the directive contains the new rule. - `execution-blocker-read.test.ts`: a retryable blocker's nextAction ends with the person-Retry text. - 54 tests pass locally for these two files. - Paid live run of `recovery.retry-exhausted` with this change and #20270 on the eval branch: the run ID will be added below. ## Risks - Low. Only text changes. Any UI that shows the blocker's nextAction shows the new sentence instead of "Try again or send a new message ...". - No schema or saved-state change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 10:53:08 -06:00
DottaandClaude Opus 5.5	90404d2033	perf(run-logs): read object-store log tails in parallel, once, and cached (#20284 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Hosted instances keep run logs in object storage as a journal of content-addressed segments and binary index nodes > - The task chat reads the last 32 KB of a run's log, aligned to a record, to show what the agent did > - A run that flushes often has many small segments. The tail read fetched each object one after another, and record alignment walked the index a second time > - On staging, one tail read took 11–12 s, and the task chat waited for it (see #20277) > - This pull request reads the tail in parallel, in one pass, and caches the objects that cannot change > - The benefit is that the first tail read is about 6.6x faster, and repeated reads (live polling, reopening a task) use only the mutable head ## Linked Issues or Issue Description What happened? `GET /api/heartbeat-runs/:runId/log?offset=…&limitBytes=32000&alignStartToRecord=true` took 11–12 s on staging for a 400 KB object-store log. Without alignment it took 6.1 s. A read from offset 0 took 2.0 s. The task chat waited for this read before it showed the run's output. Expected behavior A 32 KB tail read returns in well under a second, and a repeated read of the same tail costs about one storage round trip. Steps to reproduce 1. Use object-backed run logs (`s3_segments`). 2. Write a run log in many small flushes, as a live run does. 3. Read its last 32 KB with `alignStartToRecord=true`, and count the object GETs and how many run at the same time. Paperclip version or commit `master` at `6459488984`. Deployment mode Hosted (authenticated) deployment with S3-compatible run-log storage. Refs #20276 (measurement and plan), #20277 (chat no longer waits on a slow transcript). ## What Changed - `object-byte-journal.ts`, `readRange`: - Visits the index tree and fetches sibling objects concurrently, with at most 4 fetches for each read. - Reassembles the pieces by offset. - Cancels the remaining fetches when one fails. - The shared `objectIO` limiter still caps the process at 8 slots, and background persistence still gets freed slots first (#19176). - `object-byte-journal.ts`, `load`: - Keeps segments and index nodes in a process-wide LRU, limited to 32 MB in total and 1 MB per object. The budget counts each entry's key and a fixed 256-byte overhead with its bytes. - These objects cannot change: each key ends in the SHA-256 of its bytes, and `load` verifies that digest before it caches the object. - The mutable head is never cached. - The cache and the shared fetches are scoped to the storage provider instance. Production uses one provider in the run-log store singleton. - Concurrent fetches of the same object are shared. A shared fetch has its own controller. A reader that aborts only stops its own wait. The fetch is aborted only when no reader still waits for it, so the other readers never start a second GET for that object. - `object-run-log-store.ts`, `read`: - With `alignStartToRecord`, the first 4 KB alignment window is read in the same range as the body. - The existing 32 KB backward windows still handle longer records. - The output is unchanged. - New test `object-run-log-store-tail-read.test.ts`, written against a fake provider that counts GETs: - correct aligned and unaligned content - concurrent fetches - alignment costs at most 30 extra GETs, with no second walk of the index - a repeated read costs 1 GET (the head) - appended output is visible after cached reads - a reader that shares a fetch survives another reader's abort - a reader that joined a shared fetch stops at once on its own abort - when the reader that started a fetch aborts, no object is fetched twice - a shared fetch stops when its last reader aborts - one store's cached objects are not served to a reader of another store - a tail that starts inside a record longer than the 4 KB lookback (6 KB and 90 KB records) - the cache budget counts per-entry overhead - the shared-fetch tests hold object GETs open with a gate instead of wall-clock delays ## Verification - `cd server && npx vitest run src/services/object-run-log-store-tail-read.test.ts src/services/object-run-log-store.test.ts src/services/object-byte-journal.test.ts src/services/run-log-store.test.ts`: 4 files, 74 tests pass. The tail-read file passed 5 of 5 repeated runs. - The 18 server test files that use the journal, the run-log store or `readLog`: 381 tests pass. Two skill-materialization tests timed out under load and pass when run alone (4/4). - Red checks: - On `master`, the aligned tail read made 357 GETs with at most 1 running at a time, and a repeated read made 356 GETs. - When I removed the abort handling for shared fetches, the shared-abort test failed 3 times out of 3. - The tests for the review fixes (joiner abort, no duplicate GET after the first reader aborts, last-reader abort, per-store cache scope, cache entry overhead) fail without the fix. - An off-by-one in the long-record lookback arithmetic makes both long-record tests fail. - Benchmark: same fixture, with 30 ms latency added to each GET, a 32 KB tail of 1,200 single-record segments. \| Read \| `master` \| This PR \| \|---\|---\|---\| \| Aligned \| 11,753 ms, 365 GETs, 1 at a time \| 1,778 ms, 205 GETs, 4 at a time \| \| Unaligned \| 5,979 ms, 184 GETs \| 1,607 ms, 184 GETs \| \| Repeat aligned \| 11,686 ms, 365 GETs \| 34 ms, 1 GET \| - `pnpm typecheck` in `server` passes. ## Risks - Memory: the cache can hold up to 32 MB of decoded log bytes in each server process. Authorization still happens in the routes before any read. The cache key is the full object key, which includes the run's journal prefix. - Storage concurrency: one read now holds up to 4 I/O slots instead of 1. Background persistence still gets freed slots first. Under many simultaneous cold reads, interactive reads can reach the 32-waiter limit sooner and receive the existing retryable 503 `object_journal_io_busy`. The cache makes repeated reads cheap, which offsets this. - Remaining cost: the first read of a cold tail still walks a binary index over small segments, about 1.8 s at 30 ms per GET. A larger gain needs writer-side changes, such as wider index nodes or compaction when the log closes. Those change the persisted shape, so they are not in this PR. - No migration, configuration change, persisted-shape change or new dependency. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code with tool use (shell, browser automation, file editing). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 10:26:36 -06:00
DottaandClaude Opus 5.5	099adb2728	fix(recovery): settle a remote workspace save that stopped without finishing (#20291 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs in remote sandboxes save the workspace as snapshots. Each save is a `workspace_operations` row with phase `remote_workspace_snapshot` > - The ownership registry lists that kind as in-progress work, but it had no settle step > - When a restart cut off a failing final save, the row stayed `running` after its invocations had all stopped and its run had ended > - The retained-source fallback refuses while any workspace operation is running, so a task whose saved files could not be verified could never continue from its last save > - This pull request lets the ownership reconciler settle that row as failed > - The benefit is that the automatic fallback can continue such a task without any user action ## Linked Issues or Issue Description What happened? A native run in a remote sandbox finished its turn. Its final workspace save failed because the sandbox disk was full ("No space left on device" in the snapshot copy). A graceful restart then interrupted the retry. The run ended as `native_workspace_sync_out_retry_exhausted`. Its `remote_workspace_snapshot` row kept `status: running` for six days, with `saveWork: { activeIds: [], unknown: false, quiescedAt: <time> }` and `exportCleanup: { state: "pending" }`. The retained-source fallback checked the task and refused with `workspace_operation_running`, so the task stayed blocked. Expected behavior A save with no active or unknown invocation, whose run is terminal, is finished work. The reconciler must mark it failed, so the fallback can continue from the last committed save. A save that might still be running must stay untouched. Steps to reproduce 1. Run a native task in a remote sandbox. Make its final save fail, then restart the server during the retry, so the save's work is quiesced and the operation stays `running`. 2. Let the task's recovery refuse with an unverifiable-source failure (for example `allocation_ownership_ambiguous`). 3. The retained-source fallback refuses with `workspace_operation_running` on every check, and the task never continues. Paperclip version or commit `master` at `f7c85bca18`. Refs #20282 (the other cause of the same stuck task: a false ownership refusal). ## What Changed - `settleAbandonedRemoteSnapshotSave` (`recovery/workspace-finalize-attempt-settlement.ts`) settles a `remote_workspace_snapshot` row in `running` or `pending` only when all of these are true: - its save work is quiesced (`activeIds` is empty and `unknown` is false); - its run is terminal. It sets `status: failed`, records `abandonedSave` provenance and `lastFailure: remote_workspace_save_abandoned`, keeps `saveWork` and `exportCleanup`, writes a `workspace_operation.reconciled` activity entry, and wakes the workspace's terminal work. - The ownership registry wires it as the settle of `remote_snapshot_save`. - In the same transaction it re-arms blocked fallbacks. For every open hold whose fallback check last refused with `workspace_operation_running` and whose source run used this save's workspace, it clears the check and wakes the workspace-recovery queue. A hold that had already spent its 12 checks (no next check) continues as well. - The settle uses the save path's own `readSaveWork`, now exported, so it never settles save work that the save path would treat as invalid. - Because the row now has a `lastFailure`, `retireQuiescedSaves` retires it like any failed save when a later final save commits. A later retry of the same save boundary reopens a failed row, as before. - The ownership inventory row for this kind names the new settle. ## Verification - Red before the fix, green after: `settles a remote snapshot save that stopped without finishing after its run ended, and keeps open or live ones` (`ownership-reconciliation-report.test.ts`). Before: `settled: 0`. After: only the abandoned save settles. Saves with a live run, an active invocation, or unknown save work stay `running`. - Red before the fix, green after, end to end: `continues once the reconciler settles a final save a restart left running after its run ended` (`workspace-recovery-fallback.test.ts`). The fallback first refuses with `workspace_operation_running`, and the test then spends its check budget (12 checks, no next check), so the hold drops out of discovery. After one ownership reconciliation pass, the save is `failed` and the check is cleared. The worker then resolves the hold with `retained_source_unverifiable_continued_from_last_saved`. Without the re-arm, this test fails. - The reconciler test also covers save work that is not quiesced (`quiescedAt: null`), save work that is invalid (`quiescedAt: ""`), and a `pending` save. - `workspace-recovery-fallback.test.ts` (16 tests), `ownership-reconciliation-report.test.ts`, and the remote workspace lifecycle, save-concurrency, and snapshot-retention suites pass (80 tests, 1 environment-gated file skipped). - `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm typecheck` in `server` passed. ## Risks - A save in another process registers its invocation in `activeIds` in the same row-locked transaction before it runs. The settle locks the row and requires `activeIds` to be empty and `unknown` to be false, so it cannot fail a save that is still working. - A save whose process died with its invocation still in `activeIds` stays `running`. That case needs an owner-death proof, which is out of scope here. - No schema or API change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, read-only browser diagnostics). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 10:26:04 -06:00
DottaandClaude Opus 5.5	ef2b8d5324	fix(issues): make every Retry start work or say why (#20283 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a task's run fails, the board recovers it with a Retry button, so that button must either start work or say why it did not > - An audit of every Retry-type button found three gaps. The task Retry shown during a scheduled retry's countdown returned "accepted" but changed nothing. A stale Retry looked like success in the Inbox. "Retry now" skipped the task's read and change checks, so a member who cannot see a private task could start its retry > - This pull request makes the task Retry start a scheduled retry immediately, makes every Retry that did not start work say why, and gives "Retry now" the same authorization as the task Retry > - The benefit is that no Retry click silently does nothing ## Linked Issues or Issue Description Refs #19831. The board reported a dead Retry button. A read-only audit of every Retry button confirmed: - Task Retry during a scheduled retry was a no-op. `TaskInterruptionPresentation.canRetry` is true in `retry_scheduled`, so `ExecutionBlockerNotice` shows Retry. `requestTaskRetry` treats `scheduled_retry` as busy and inserts a `coalesced` wake pointing at that run. `resumeTaskRetry` reads only `deferred_issue_execution`, and dispatch uses coalesced user wakes only to carry comments. Nothing promoted the run: the API answered 202 `accepted: true` and the countdown did not change. - A stale Retry looked like success. `accepted: false, stale: true` comes back as HTTP 200. The Inbox ignored `accepted` and opened the task as if work had started. The task page showed no reason. - "Retry now" checked less than the task Retry. It ran only `assertBoard` and `getAccessibleResource`, skipping `assertIssueReadAllowed` and `issue:mutate`. A company member who cannot read a private task could promote that task's scheduled retry. - "Retry now" said nothing for some outcomes. It toasted only `promoted` and `gate_suppressed`. ## What Changed - `server/src/services/task-retry.ts`: when the Retry coalesces into the task's latest run and that run is `scheduled_retry` (with no other Retry already active), the receipt returns `scheduledRetryRunId`. - `server/src/routes/issues.ts`: - `POST /issues/:id/retry` then promotes that run through the existing `retryScheduledRetryNow`, the same path "Retry now" uses. It logs `issue.scheduled_retry_retry_now` with `source: "task_retry"` and returns `scheduledRetry: { outcome, message }`. - `POST /issues/:id/scheduled-retry/retry-now` now runs `assertIssueReadAllowed` and `decideIssueAccess("issue:mutate")`, matching `/retry`. A private task the member cannot read now answers 404. - `ui/src/hooks/useTaskRetry.ts`: new `taskRetryNotStartedReason`. When a Retry did not start work (stale, or a scheduled retry that is gate-suppressed, waiting, or gone), the hook returns that reason as its error. `ExecutionBlockerNotice` and `StalledDependencyNotice` already display it. - `ui/src/pages/Inbox.tsx`: a task Retry that did not start work shows the reason in the error toast and does not navigate. - `ui/src/hooks/useRetryNowMutation.ts` and `IssueScheduledRetryCard.tsx`: every "Retry now" outcome now shows its headline and message, including `no_scheduled_retry` on the card. - `ui/src/api/issues.ts`: `retryTask`'s response type includes `scheduledRetry`. ## Verification Red/green (each new test fails against master and passes with this PR): - `server/src/__tests__/issue-scheduled-retry-routes.test.ts`: - "starts a scheduled retry now when the task Retry is clicked during its countdown": 202 with `scheduledRetry.outcome: "promoted"`, the same run becomes `queued`, and no second retry run is created. - "refuses Retry now on a private task the member cannot see, like the task Retry": on master "Retry now" answers 200 `promoted`; now both routes answer 404 and the run stays `scheduled_retry`. - The seed now adds the board user's membership, which the stricter route needs. All 33 tests pass. - `ui/src/hooks/useTaskRetry.test.tsx`: a replaced attempt, a gate-held scheduled retry and a waiting scheduled retry each show their reason; a promoted one shows nothing extra. - `ui/src/pages/Inbox.test.tsx`: an Inbox Retry that did not start work (stale, or gate-held) toasts the reason and does not navigate. - `ui/src/components/IssueScheduledRetryCard.test.tsx`: "explains when there was no scheduled retry left to start". Other Retry buttons: the chat failed-run Retry already reports stopped and failed retries (`agents.test.ts`). Agent lifecycle "Retry now" and cost "Retry accounting" start no task work and already surface their API errors. Regression and static checks: - UI: 7 files, 168 passed. Server: task-retry and scheduled-retry suites, 149 passed. `issue-agent-mutation-ownership-routes.test.ts` passed. - `maintenance-server-routes.test.ts` "assembled maintenance server production wiring" fails identically on master on this Mac (`ensure-build-deps`). - `tsc --noEmit` passes for `server` and `ui/src`. `pnpm check:token-gates` is clean. ## Risks - A task Retry during a scheduled retry now starts the run immediately, through the existing "Retry now" promotion gates (pause, budget, cleanup waits). A gate that holds it is reported, not bypassed. - "Retry now" is stricter: a member without read or change access to the task gets 404 or 403, as on `/retry`. - No schema change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 10:25:21 -06:00
Dotta	5c73633a8f	fix(workspaces): preserve remote recovery holds during online import (#20298 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Online workspace import preserves old controller-local files while unrelated remote work continues. > - A task can have an old local checkout and a separate durable remote workspace. > - A terminal remote run can retain a file-recovery hold on that task. > - The legacy source checker rejects remote references, so that hold blocks independent local imports. > - This pull request proves that the hold belongs only to the separate remote source and fences its finalizer during import. > - Old files can migrate while the remote recovery obligation stays intact. ## Linked Issues or Issue Description Refs #19873 and #19854 for the existing cohort and source-verification paths. Searched related online migration and remote recovery PRs; no duplicate fix was found. What happened? Online import rejects a legacy Git checkout with `online_workspace_source_scope_unproven` when a same-task failed run has a prepared reference to a separate durable remote workspace. The legacy run reader rejects remote references before the hold can be classified. Expected behavior Independent local files can import under their source fences. The separate remote workspace, dirty writer, allocation, accepted result and recovery hold must stay intact. Steps to reproduce 1. Create related local Git workspaces. 2. Record a terminal failed native run and an exhausted file-recovery hold for a separate durable remote workspace on the same task. 3. Import the local leaf online. The old checker refuses before capture. Paperclip version or commit Reproduced on master at `052c638b519ebb15f81e4fcd1a96adf6cd3b2f10`. Deployment mode Managed server with PostgreSQL and remote Runner execution. ## What Changed - Add a distinct remote-only preservation proof. Keep remote runs excluded from legacy export sources. - Use the canonical source inspector and prepared-reference matcher. Bind the run, finalizer, action, task, accepted result, authority, allocation and prior committed snapshot. - Acquire the held run's finalizer fence through the existing sorted lock path. Existing source-proof reinspection rejects changes before publication. - Preserve previous-release prepared references with an older baseline snapshot. Do not require that snapshot's epoch or source run to equal the current run. - Add real PostgreSQL import, authenticated cold restore, preservation, refusal, lock-contention and publication-race regressions. Document the behavior. ## Verification - Genuine baseline failure on unchanged product: the real importer refuses with `online_workspace_source_scope_unproven`. - Final focused regression file: 23 tests passed in 13.19 seconds through the canonical isolated PostgreSQL wrapper on Node 24.22.2. - 24 existing affected migration/recovery tests passed against the same final production bytes. Another 83 tests were explicitly excluded by the focused selection; no full-suite pass is claimed. - The positive imports local bytes, confirms the remote finalizer fence is held during upload, leaves all remote records unchanged, parks the original repository and authenticates/restores the archive. - Refusal and race cases cover malformed, foreign, same-local and mixed references; active owners/writers; changed allocation, result, task, action, authority and baseline; and finalizer contention. - Server TypeScript check, module-boundary check and `git diff --check` passed. - An intermediate run had one incorrect assertion that expected an explicit empty legacy-source array. The canonical journal omits that empty field. The final assertion accepts absence and still forbids any exported remote source. Final relevant results are 23 new plus 24 existing passes, rather than a claim that the earlier invocation passed in full. - Required CI passed on head `6a278e717db2b5f614b0a595f99bfd0fb5430227`: 55 successful checks and 4 intentional skips. [CI run](https://github.com/paperclipai/paperclip/actions/runs/42650025033). Greptile reviewed that head at 5/5 with no actionable findings or open inline/review comments. Hosted continuation remains pending normal delivery. ## Risks - A false disjointness proof could admit an unsafe local import. Strict company/task/source binding, the canonical inspector, finalizer fencing and publication revalidation retain refusal on uncertainty. - This does not recover or settle remote files. Dirty authority and the original recovery obligation remain unchanged. - No schema, public API, archive format or saved-run identity changes. Existing persisted state carries forward. ## Model Used OpenAI Codex, GPT-6.1 Sol (`gpt-6.1-sol`), with reasoning, repository tools and code execution. The runtime does not expose the context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge	2026-11-09 10:13:41 -06:00
DottaandClaude Opus 5.5	d10c6e0e8b	fix(chat): hold a chat message at the agent's daily limit instead of dropping it (#20292 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agent Chat sends each person's message to the conversation agent through a durable outbox (`deliverConversationComments`). The outbox keys each wake receipt as `conversation-comment:<commentId>`. > - The outbox treats any receipt with that key as delivered. Queue admission writes a `skipped` receipt when the agent reached its daily run or cost limit. > - So a chat message sent after the agent used its daily allowance was consumed silently. No run started, the chat showed nothing, and nothing resumed it. > - A person's message must never be silently dropped. The daily limit is an operator control, so the message must wait honestly, not bypass the limit. > - This pull request makes the outbox hold the message at the daily limit, like it already does for a paused agent, and resume it when the limit resets. > - The benefit is that a chat message at the daily limit is saved, shown with its reason, and sent automatically at 00:00 UTC. ## Linked Issues or Issue Description No public issue exists. Inline bug report: What happened In a chat eval, the conversation agent (CEO) had `maxDailyRuns: 3`. After three chat turns and a separate failed-model phase on another agent, the person sent a fourth message. The comment was saved (POST 201), but no CEO run started. The conversation stayed `active` with no `executionRunId`. The ownership reconciler later reported `I7_visible_next_action`. The server log shows no wake error. Queue admission had written a `skipped` receipt (`heartbeat.daily_run_limit`) under the outbox key, so the outbox counted the message as delivered. Expected behavior A chat message that reaches the agent's daily limit is not consumed. It stays saved, the chat shows why it waits, and it is sent automatically when the limit resets. The daily limit still applies: the message does not start a run early. Steps to reproduce 1. Set a conversation agent's `runtimeConfig.heartbeat.maxDailyRuns` to N. 2. Send N chat messages and let each turn finish. 3. Send one more chat message. 4. See that no run starts, the chat shows no waiting state, and the message is never answered (a `skipped` wake receipt with reason `heartbeat.daily_run_limit` exists for `conversation-comment:<id>`). ## What Changed - `server/src/services/agent-conversations.ts`: add `conversationDeliveryHold`, which applies the same daily run/cost limit check as queue admission. The outbox returns `deferred` at the limit and writes no receipt, so the message stays owed. It reports the next UTC day start through a new optional `onDeferredDeadline` option. - `server/src/services/recovery/service.ts`: stranded-work recovery passes its `onDeferredDeadline` to the outbox, so the held message is retried when the limit resets. - `server/src/services/heartbeat/queue.ts`: if the limit is reached while an outbox wake is in admission (a race), admission throws a 409 with `reason: heartbeat.daily_*` for `conversation-comment:` keys instead of writing a `skipped` receipt. The outbox already maps a matching 409 to `deferred`. Other wakes keep the current skip behavior. - `server/src/routes/issues.ts` (`GET /issues/:id/queued-comments`): when a chat has no queue receipt but has messages held by the daily limit, return them as a `deferred` queue with `executionWait` (reason and message). Edit, send and discard stay off because the outbox owns delivery. - `ui/src/components/task-chat/TaskChatQueuedMessages.tsx`: show the `executionWait` message for such an outbox-held queue (no queue id and no target run). Receipt-backed queues keep the current behavior and do not duplicate the task status. - Tests: three real-database tests in `agent-conversation-delivery.test.ts` (hold and resume after the limit resets, recovery deadline, and a limit reached during real `heartbeatService.wakeup` admission) and one UI test. ## Verification - `cd server && pnpm exec vitest run src/__tests__/agent-conversation-delivery.test.ts`: 10 passed. Without the source change, the three new tests fail (`expected 'delivered' to be 'deferred'`; the real queue path writes the skipped receipt). - `cd server && pnpm exec vitest run src/__tests__/issue-recovery-conversations.test.ts src/__tests__/agent-conversations.test.ts src/__tests__/heartbeat-undispatched-execution-wait.test.ts src/services/heartbeat/queue.test.ts src/__tests__/agent-chat-messages.test.ts`: 215 passed. - `cd ui && pnpm exec vitest run src/components/task-chat/TaskChatQueuedMessages.test.tsx`: 36 passed. - `pnpm exec tsc --noEmit -p server/tsconfig.json`: clean. `node scripts/check-token-gates.mjs`: clean. - Manual: give a chat agent `maxDailyRuns: 1`, send two messages. The second message shows in the queued messages bar with "<agent> reached its daily run limit. Your message is saved and sends automatically when the limit resets at 00:00 UTC." ## Risks - Behavior change for chat only: a chat message at the daily limit now waits in the outbox instead of being consumed. It is sent after 00:00 UTC. If an operator raises the limit, the message is sent on the next recovery pass for that chat; the UI message does not promise an earlier send. - Saved state from the previous release: no persisted shape changes. Messages that an earlier release already consumed with a `skipped` receipt stay consumed. They are not replayed later, which avoids sending old messages without notice. - The queued-comments read adds one agent read and one daily-limit count for a chat with no queue receipt. This is the same query that admission runs. - The recovery deadline is in-process; instance-sleep scheduling does not get a new deadline from this change. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), agentic coding with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 10:03:28 -06:00
DottaandClaude Opus 5.5	87e755ce8e	fix(recovery): verify cross-task warm sandbox handoffs and say plainly when recovery refuses (#20282 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs in remote sandboxes keep a warm sandbox and hand it to the next run through a chain of reusable leases > - Acquisition binds that chain to the execution workspace, so a task can inherit a warm sandbox from an earlier task in the same workspace > - Saved-files recovery verifies the same chain, but it required every earlier lease, run, and finalization to belong to the recovering task > - A valid chain that started with another task therefore always failed with `allocation_ownership_ambiguous`. That code is `service_required`, so recovery stopped after one attempt, and each board Retry repeated the same refusal > - This pull request makes the chain check match the acquisition rule, and it checks each predecessor against its own task > - It also makes the board say plainly when recovery refuses the saved state, because "failed after automatic retries" was not true and Retry looked like it did nothing > - The benefit is that a task blocked with "Saving the agent's files failed after automatic retries" can recover its saved files, and a refusal that Retry cannot change is shown with its reason ## Linked Issues or Issue Description What happened? A native run in a remote sandbox finished its turn, and its result was accepted. The copy of its files back to the host failed and was scheduled to retry. A graceful server restart then interrupted the retry, so the run settled with `native_workspace_sync_out_retry_exhausted`, and the task became blocked. Every saved-files recovery generation then failed on its first attempt with `allocation_ownership_ambiguous` (`service_required`). The board showed "Saving the agent's files failed after automatic retries. Retry to try again". Each Retry replaced the generation (`issue.recovery_generation_replaced`), and the new generation failed in the same way within one second. To the user, Retry did nothing. The sandbox lease chain was valid: same agent, same execution workspace, same sandbox, and matching handoff receipts on all eight leases. But its first three leases belonged to an earlier task in the same workspace. Expected behavior Recovery must accept a warm sandbox chain that acquisition accepted. Acquisition binds reuse to the execution workspace, and it requires the same task only when the lease has no workspace. Recovery must check each predecessor run and finalization against that predecessor's own task. It must still refuse a chain whose records disagree or that has no shared workspace. Steps to reproduce 1. Run a native task in a remote sandbox with `reuse_by_environment`, with an execution workspace that another task also uses. 2. Run the other task in the same workspace first, so the sandbox lease is handed from that task to this one. 3. Make the last run of the second task fail to copy its files back (for example, restart the server during the copy-back retry). 4. Wait for saved-files recovery, or click Retry. Recovery fails with `allocation_ownership_ambiguous`, and Retry repeats the failure. Paperclip version or commit `master` at `a3a25ce6c9` (the same code as the 2026.1106.0 staging build for every file this change touches). ## What Changed - `retiredReusableLeasePredecessors` (`environment-lease-handoff.ts`) requires the same task only when the lease has no execution workspace. This matches the binding that `acquireLease` uses when it replaces a reusable lease. - `inspectWorkspaceRecoverySource` (`workspace-recovery-source.ts`) checks each predecessor run and its finalization against the predecessor lease's own task, not the recovering task. A predecessor from another task is accepted only when both leases have the same execution workspace. All other checks stay: same agent, terminal run, no live controller or lease owner, a settled finalization, and a saved snapshot at an older epoch on the same allocation, workspace, and remote directory. - The other callers of the chain walk (remote termination resolution and retired backup restore) get the same rule. - The test fixture `warmHandoffSource` gets an option that starts the chain with an earlier task in a shared workspace. - Honest copy for exhausted recovery (`exhaustedSystemRecoveryAction`): - A deterministic refusal (`service_required`) now says "Paperclip refused to recover the saved files", names the reason in plain words and the code, and says that Retry runs the same check again and succeeds only after Paperclip is updated. - Other exhausted recovery says "stopped after N attempts" only when there were several attempts. One attempt says "failed". - Continuation (`explicit-native-continuation.ts`) reports a new wait reason, `saved_work_recovery_refused`, when the exhausted recovery is a deterministic refusal. The task presentation shows its own message for it. `saved_work_recovery_failed` no longer claims automatic retries. - Retry stays available for a refusal. It is the action that runs recovery again after an update. After each Retry, the board shows the same refusal and its reason, not an unchanged generic notice. ### Recovery refusal classes \| Retry class \| Failure codes \| Automatic recovery \| Board Retry \| \| --- \| --- \| --- \| --- \| \| `retry_transient` \| `provider_unavailable`, `export_interrupted` and other transient codes \| Retries with backoff, up to 5 retries in 30 minutes \| Starts a new recovery round \| \| `recheck_evidence` \| `agent_stop_unconfirmed`, `workspace_writer_active`, `effect_unresolved`, `copyback_stop_unconfirmed` \| Checks the evidence again later \| Starts a new round (the copy-back stop case has its own wait text) \| \| `use_alternative_strategy` \| `historical_source_missing`, `source_unavailable` (no recorded source) \| Stops. The retained-source fallback can continue from the last save \| Starts a fresh attempt \| \| `service_required` (deterministic) \| `allocation_ownership_ambiguous`, `source_binding_invalid`, `snapshot_verification_failed`, `archive_unsafe`, `source_unavailable` (recorded source gone) \| Stops after one attempt. The same saved state gives the same result \| Runs the same check again. The board now says this and names the reason \| ## Verification - Red before the fix: `recovers saved work after 1\|2 warm handoffs that began with another task in the same workspace` failed with `expected 'allocation_ownership_ambiguous' to be null`. - Green after the fix: `npx vitest run src/services/workspace-recovery-warm-handoff.test.ts src/services/workspace-recovery-source.test.ts` passed (60 tests). - Safety cases that stay fenced: a predecessor lease that names a different task than its run (`lease_task`), and a cross-task predecessor with no shared workspace (`unshared_workspace`). The existing `task` fault (another task with no workspace) also stays fenced. - Sibling and retry suites passed: environment runtime and orphan sandbox, cancelled workspace preservation, workspace recovery fallback, task retry, remote workspace lifecycle, explicit native continuation, native cleanup admission, and remote retry admission. - Copy, red before and green after: `system-recovery-state.test.ts` (`states a deterministic refusal plainly and never claims retries that did not happen`), and the end-to-end `names a deterministic saved-files refusal behind Retry when no fallback can continue, and another Retry shows it again` in `workspace-recovery-preparation-gates.test.ts`. On the old code the wait reason was `saved_work_recovery_failed`. With the change it is `saved_work_recovery_refused`, with the refusal reason, for both Retry clicks. - Updated expectation: the board handoff in `workspace-recovery-retry-copyback.test.ts` now expects the refusal text for its `source_binding_invalid` source (79 tests pass). - `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm typecheck` in `server` passed. - I compared the real incident's lease chain with every predecessor condition through read-only diagnostics. All eight leases match on agent, sandbox, workspace, and remote directory, and each predecessor run saved a snapshot at an older epoch. ## Risks - Recovery now trusts a predecessor from another task in the same execution workspace. Acquisition already allows that handoff, and recovery still requires the predecessor's own run and finalization to be settled and saved. A workspace is the unit that owns the files, so this does not let one workspace read another workspace's files. - I could not read one predecessor's finalization row in the real incident. If that row is not in a settled phase, recovery still refuses with `allocation_ownership_ambiguous`, which is the old behavior. - `snapshot_verification_failed` is labeled `service_required` at its 19 call sites, so it now shows the refusal text. If one of those sites is really transient, the text says "Retry runs the check again", which is still true. - A second cause kept the same task stuck: a remote snapshot save that a restart interrupted stayed `running`, and that blocks the retained-source fallback. That fix comes in a separate pull request. - No schema or API change. The new wait reason is a new string in an existing field. The UI shows the message as it receives it. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, read-only browser diagnostics). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 09:53:37 -06:00
Nicky LeachandPaperclip	052c638b51	fix(environments): honor stopped-task proof after warm preparation (#20289 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A warm plugin Runner retains its task for later runs. > - A follow-up can find that the retained task has stopped. > - The resume path already asks settle to check status before it sends stop. > - A saved startup marker disabled this order, even after warm preparation had finished. > - This pull request applies the same stop-proof check to all observed-stop releases. > - The next run skips a repeated stop when status proves both execution and required preparation have stopped. ## Linked Issues or Issue Description Fixes: #20224 Refs #20203 ## What Changed - Remove the startup-marker exclusion from the existing status-first path. Keep the existing shared confirmation check. It still requires `preparationStopped: true` for startup work. - Keep stop followed by status when the first status lacks the required proof. Keep the startup drain, credential writeback, complete, and cleanup failure handling. - Add retained-task coverage with the startup metadata that warm preparation saves. Cover a startup cleanup whose first status lacks preparation proof. - Update the plugin task lifecycle contract. ## Verification - Local macOS, source base `3e8744fa91`: from `server`, `pnpm exec vitest run src/__tests__/environment-task-runtime.test.ts src/__tests__/plugin-warm-tasks.integration.test.ts src/__tests__/plugin-task-credential-writeback.test.ts` passed: 3 files, 82 tests. The integration tests use real embedded PostgreSQL and controlled plugin results. - After restoring the fix, the two affected files passed again: 73 tests. - Independent parent verification on commit `19f720ff68`: the two files filtered to the changed startup-proof and retained-task cases passed (4 tests). Independent source review found no actionable issue. - Baseline proof: with only the source condition restored to upstream, the new retained-task test with saved startup metadata fails because it sees an extra `stop`. The case without that metadata passes. The source fix was then restored. - The retained-task regression asserts exactly `environmentResumeLease`, `status`, `complete`, and `environmentAcquireLease`, with both task operations bound to the old task. The startup test also proves that missing preparation proof sends `stop` and a second `status`. - Test setup: `node scripts/agent-preflight.mjs --skip-rust-debug --explain` installed dependencies, then stopped because Cargo is unavailable on this Mac. Initial test collection failed because the plugin SDK output was absent. `pnpm --filter @paperclipai/plugin-sdk build` built that required dependency and the shared package; the targeted test run then passed. - Standard Greptile reviewed commit `19f720ff686058b4dd6b34f27cdfb9c8ff60e50a` at 5/5 with no review threads. [Review](https://github.com/paperclipai/paperclip/pull/20289#issuecomment-6743829614). - Full local checks were not run, as requested. [GitHub PR workflow 42644023868](https://github.com/paperclipai/paperclip/actions/runs/42644023868) passed on final commit `19f720ff686058b4dd6b34f27cdfb9c8ff60e50a`. It supplied broad typecheck, build, and test coverage. The separate Snyk status passed. Optional Storybook checks were skipped. - The change removes one provider stop call on the reported path. The issue's measured eight calls would become seven if other calls stay the same. The earlier five-call target is not claimed. No deployed latency measurement is claimed. ## Risks - A provider that cannot prove preparation stopped still gets stop and status. That fallback can cost one extra status call, as the existing status-first contract already allows. - A status failure keeps cleanup pending and blocks replacement. No termination proof or credential handoff is skipped. - No schema, API, dependency, or timer change. No added work on successful task reuse or ordinary task dispatch. - Live provider acceptance remains unrun. The local evidence checks orchestration and stop-proof handling. ## Model Used - OpenAI Codex, GPT-6 family, with reasoning, repository inspection, shell tools, and code execution. The exact serving model ID and context window are not exposed by this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-09 07:44:19 -08:00
DottaandClaude Opus 5.5	dc9ad17515	perf(storage): time workspace snapshot save and restore steps and count storage retries (#20280 ) ## Thinking Path - Cloud durable storage is a major source of failures and slowness on staging. For example, a staging task failed when saving files after automatic retries. - Before measuring or fixing it, we need to see where storage time goes. Today a saved-workspace restore before the turn appears only inside the coarse `workspace_environment_realize` phase. - After the turn, the save's encrypt, put, verify and multipart throttle backoff have no timing at all. Only `captureMs`, `transferMs`, `verifyMs`, `uploadMs` and `publishMs` exist, in `saveTiming`. - Storage retries happen in the SDK and in the multipart uploader, invisibly. - This adds the measurement first, so the storage-mode chaos waves (MinIO behind a fault proxy) can report save and restore p50/p95 and point at the dominant wait. ## Linked Issues or Issue Description What happened? Durable-storage saves and restores were opaque in the run log and save timing. A slow or repeatedly retried storage call could not be told apart from slow capture, encryption or sandbox work. Steps to reproduce 1. Run a remote task with S3-compatible durable storage, so it saves after the turn and restores on the next run. 2. Inspect the run log and the `remote_workspace_snapshot` operation's `saveTiming`. The restore's download, decrypt and verify, and the save's put, verify and throttle backoff, have no timing. Expected behavior Each storage step has a duration, and storage request retries and backoff are counted. ## What Changed - `server/src/storage/request-stats.ts` (new): an async-local collector of storage request statistics. Counts only: - `requests`, `attempts` and `failedRequests`; - `retryDelayMs` (SDK waits); - `parts`, `throttled` and `backoffMs` (the multipart uploader's waits). - `server/src/storage/s3-failure.ts`: `withS3FailureDiagnostics`, which wraps every S3 send, records each request's SDK `$metadata.attempts` and `totalRetryDelay`, on success and on failure. - `server/src/storage/s3-multipart.ts`: records each uploaded part and each resend wait, flagged throttled when the backend returned 503, 429 or SlowDown. - `server/src/services/remote-workspace-snapshot-storage.ts`: - Each step is a `storage.snapshot.` run phase: - save: `prior_attempts`, `encrypt`, `digest`, `put`, `verify`; - restore: `download`, `decrypt`, `verify_plaintext`, `publish`. - A caller can pass a `timing` record that receives the step durations, the bytes and the request statistics. - `server/src/services/remote-workspaces.ts`: - The save passes a timing record, kept in `saveTiming.storage`. - The sandbox unpack during a restore is its own phase, `workspace.snapshot.sandbox_restore`. - `execution-preparation-timing.ts`: the new phase names. - `doc/run-log-events.md`: a new "Durable storage timing" section. Because the restore runs inside run preparation, its steps of 50 ms or more become `run.performance.span` rows with no extra plumbing. This is run-log and instance-database data only; there's no Telemetry change. ## Verification - `server`: `storage-s3-multipart-backoff`, `storage-s3-multipart`, `storage-s3-multipart-throttle`, `storage-s3-failure-diagnostics`, `remote-workspace-snapshot-storage` and `execution-preparation-timing` tests: 86 passed. They include: - collected stats for a throttled multipart upload: 12 requests, 7 failed, 7 throttled, 35.5 s backoff and 3 parts; - a save and restore that fill every step duration and the byte count. - `remote-workspaces.integration.test.ts`: 112 passed. Its `saveTiming` assertion now includes `storage`. - Server typecheck: no errors in the touched files. ## Risks - Overhead:* one async-local lookup per S3 request, plus timestamps per snapshot step. - Collector scope: the collector only exists inside a snapshot save or restore. Other storage callers are unaffected; they record nothing. - `saveTiming` gains a `storage` object. Readers that compare it exactly (one test, updated) will see the new field. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`, Anthropic), 1M context window, tool use and code execution, via Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 09:41:53 -06:00
DottaandClaude Opus 5.5	3ef45fdd92	fix(recovery): let a new task wait for its manually paused assignee (#20281 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A person can pause an agent. Tasks assigned to a paused agent wait until the person resumes it. > - Stranded-task recovery looks for assigned work that cannot continue and moves it to `blocked` for the board. > - A task created for a manually paused agent has no run yet. Recovery saw an assignee that is not invokable and moved the new task to `blocked` a few seconds after creation. > - The person paused the agent on purpose. The blocked task and its "the board must choose the next action" comment fight that decision, and the person must unblock the task by hand after resume. > - This pull request lets new `todo` work with no run wait while a manual pause lasts. The existing assigned-todo dispatch starts it when the agent is resumed. > - The benefit is that work queued for a paused agent starts on resume, as the pause promises. ## Linked Issues or Issue Description What happened A chat evaluation pauses an agent (Morgan) and then runs `paperclipai issue create ... --assignee-agent-id <Morgan> --status todo`. The CLI returns the task with status `todo`. Within about five seconds the task changes to `blocked`, with the system comment "Paperclip cannot safely continue automatic recovery because the original assignee is not invokable. The source assignment is unchanged and the board must choose the next action." The server log shows only the expected 409 "Agent is not invokable" for the assignment wake. Expected behavior The task stays `todo` while the person keeps the agent paused. When the person resumes the agent, the task starts once. A comment and a `blocked` status are not added. Steps to reproduce 1. Pause an agent from its page. 2. Create a `todo` task assigned to that agent. 3. Wait for the stranded-task recovery sweep. See that the task is `blocked` with the comment above. ## What Changed - `server/src/services/recovery/service.ts`: in `reconcileStrandedAssignedIssues`, skip a `todo` task with no run when its assignee is paused with a `manual` lifecycle hold. The existing rule above it already skips work whose run the pause cancelled. Other pause causes, such as a budget hold, keep the existing escalation. - `server/src/__tests__/heartbeat-paused-assignee-new-task.test.ts`: a PostgreSQL test. The task stays `todo` with no comment and no run during two recovery sweeps. After resume it starts exactly once and finishes. A budget-held assignee still escalates. ## Verification - `cd server && npx vitest run src/__tests__/heartbeat-paused-assignee-new-task.test.ts`: 2 passed. Without the fix, the first test fails with `expected 'blocked' to be 'todo'`. - `npx vitest run src/__tests__/issue-recovery-actions.test.ts src/__tests__/heartbeat-refused-assignment-wake.test.ts`: passed. - `src/__tests__/issue-tree-pause-resume-run.test.ts` had timing failures on this low-memory machine both with and without this change (different tests each time). Those tests do not pause agents, so this change does not reach them. CI runs them on this branch. ## Risks - Low risk. The skip needs all of these: the agent is paused, it has a `manual` hold, the task is `todo`, and the task has no run. Work that already ran keeps the current rules. - No persisted state changes shape. State saved by the previous release carries forward: a task that the previous release already moved to `blocked` stays `blocked`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Related: #20143 (a pause's stop never blocks the task behind an incident), #9998 (open: reject paused agents as assignment targets). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 09:26:20 -06:00
DottaandClaude Opus 5.5	6db0bb9776	fix(chat): tell agents which CLI commands a person runs in their own terminal (#20267 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - People talk to agents in chat. The chat guidance in `server/src/services/agent-conversations.ts` tells every chat agent how to answer. > - People often ask a chat agent for a terminal command: create a task, download a plan, inspect or stop a run, or point the CLI at a company. > - The guidance did not name the Paperclip CLI. Agents guessed: they did the work themselves, wrote `curl`, used placeholder company ids, or invented an API address. > - The person then has to fix the command, or gets a file in the agent's workspace instead of on their own machine. > - This pull request adds one guidance paragraph. It names the supported `npx paperclipai` commands, asks for real ids from the conversation, and asks for the person's own Paperclip URL as a placeholder. > - The benefit is a command the person can run as it is, with the right company and the right actor. ## Linked Issues or Issue Description What happened Three chat evaluations (cli-operations) asked for terminal commands. "Show me how to point my terminal at the Acorn company." got `context set` with `<acorn-company-id>` and an invented `http://localhost:3100`, although the agent can read its company id. "Download this task's plan from the terminal as a file." got a file saved in the agent's own workspace and an attachment link, with no terminal command. "Show me the command to inspect Morgan's last run and stop it if it is still running." got `curl` with environment variables that exist only in the agent's runtime. Expected behavior The agent gives `npx paperclipai` commands that the person runs in their own terminal. The commands use real ids that the agent can read in the conversation. The Paperclip address is the placeholder `<your Paperclip URL>`. The agent says which commands need a board user. Steps to reproduce 1. Open a chat with an agent. 2. Send "Download this task's plan from the terminal as a file." 3. See that the agent saves the file itself and gives no terminal command. ## What Changed - `server/src/services/agent-conversations.ts`: add a chat guidance paragraph for terminal and CLI requests. It names the supported commands for task creation, document download, run read and cancel, and context profiles. It asks for one command per line, real ids, and the `<your Paperclip URL>` placeholder. - `cli/src/__tests__/chat-cli-guidance.test.ts`: a new test reads the guidance and resolves every `npx paperclipai` command against the registered commander commands. It fails on an unknown command, flag or missing argument. - `server/src/__tests__/agent-conversations.test.ts`: check that the paragraph is in the directive and in the chat prompt. ## Verification - `cd cli && npx vitest run src/__tests__/chat-cli-guidance.test.ts`: 3 passed. - `cd server && npx vitest run src/__tests__/agent-conversations.test.ts src/__tests__/board-access-denial.test.ts`: 80 passed. - The chat evaluations `cli-operations.choose-cli-context`, `export-document-cli` and `inspect-run-cli` run with this commit on an evaluation branch. Results are added in a comment. ## Risks - Low risk. The change adds prompt text and tests. Routes, permissions and responses do not change. - The chat prompt is about 300 words longer. - If a CLI command or flag changes, the new CLI test fails until the guidance changes too. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Related: #20029 (terminal sign-in guidance in the API catalog). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 09:26:13 -06:00
Pavel-GuseynovandClaude Opus 5.5	4d297b998a	fix(server): give remote MCP tool calls a 30 second default deadline and record stream failure evidence (#19768 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The tool gateway calls remote MCP servers over Streamable HTTP, and a server can answer `tools/call` with a `text/event-stream` body > - Such a server can send progress notifications before the response, and a slow tool can take more than 10 seconds > - The gateway aborts a remote MCP `tools/call` that uses the default deadline after 10 seconds, the general tool default > - When the call fails, the call evidence does not show how far the stream got, or whether the cause was the deadline or a broken transport > - This pull request gives remote MCP calls a default deadline of 30 seconds, the default response timeout of the remote HTTP client, and records the failure cause and the stream progress > - The benefit is that slow remote tools complete, and an operator can tell a deadline from a peer reset in the call evidence ## Linked Issues or Issue Description Refs #10147. That issue reports that a named MCP gateway aborts remote tools after 10 seconds and ignores the connection timeoutMs. This pull request raises the default deadline of a remote MCP `tools/call` to 30 seconds. It does not make the named gateway read the connection timeoutMs. Related: #11910 (open) keeps a connection callable after one timed-out tools/call. On current master a timeout and a JSON-RPC response error already keep the connection healthy, and this pull request keeps that policy. I searched pull requests and issues for "event stream tools/call", "SSE MCP tool call hang", "remote MCP timeout stream", "mcp text/event-stream" and "tool_timeout remote MCP". The only related open pull request is #11910. ## What Changed - `server/src/services/remote-http-fetch.ts`: the default response timeout is exported as `DEFAULT_REMOTE_HTTP_RESPONSE_TIMEOUT_MS` (30,000 ms, unchanged value). - `server/src/services/tool-gateway.ts`: - A remote MCP `tools/call` that uses the default deadline now gets `DEFAULT_REMOTE_HTTP_RESPONSE_TIMEOUT_MS` instead of the general tool default of 10 seconds. The Railway `run-command` budget and an explicit `timeoutMs` keep their values. - The call evidence records the response status, content type and request id before the body is read. A body that fails partway records the bytes received so far as its size. - A failed call records `failureKind` (`invocation_timeout`, `protocol_failure` or `transport_failure`), the deadline and whether it was aborted, the stream progress, and the error with its cause chain. Each string in the chain is redacted with `redactSensitiveText`, and a cyclic cause chain is cut with a "[Circular cause]" marker. - A deadline abort is a 504 `tool_timeout` and starts a fresh MCP session on the next call, as before. The connection health policy of master does not change. - A deliberate stop of a call (`elicitation_required`, `provider_interaction_required`, `oauth_insufficient_scope`, `oauth_refreshed_retry_required`) keeps its own reason when the deadline passes while it is recorded, as on master. An error that is already a gateway error also gets a failure kind: `tool_timeout` is `invocation_timeout`; a malformed, invalid or oversized response is `protocol_failure`; and `remote_http_*` (for example a DNS failure) and `mcp_remote_fetch_failed` are `transport_failure`. - A non-success response body is read as a stream, so a body that fails partway records the bytes that arrived. The reader stops at the size limit and cancels the body, as the `tools/call` reader does. #19785 makes the same reader change for the context path. - `server/src/services/mcp-http.ts`: `readMcpHttpResponse` takes an `onProgress` callback. It reports bytes, chunks and events received, whether the matching response arrived, and whether the stream ended. As on master, the read returns as soon as the response with the matching id arrives. - Tests: - `server/src/__tests__/mcp-http.test.ts`: a partial event followed by a peer error keeps the progress that was consumed, and releases the reader. - `server/src/__tests__/tool-gateway.test.ts`: six scenarios run against a remote server that streams one progress event first: slow success after 12 seconds, the default deadline, an explicit 1,500 ms deadline, a peer reset, a cyclic error cause, and a primitive error value. - `server/src/__tests__/tool-gateway.test.ts`: three regression tests: a tool question sent after the deadline passed keeps `elicitation_required`; a DNS failure records `transport_failure`; and a 503 body that fails partway records the bytes that arrived. All three fail on `717e6c77c` and pass on this head. - `server/src/__tests__/tool-gateway-service.test.ts`: when the call deadline passes during a forced Vercel token refresh, the call keeps the 409 `oauth_refreshed_retry_required` stop. `tool-gateway.ts` adds this reason to the deliberate stops, so the deadline does not change it to a 504 `tool_timeout`. ## Verification Head SHA: `25c6e703d1ded895989e9d2fcd6d6352a4f851f5`. It is the previous head `7836447f2` plus commit `25c6e703d`, which keeps the `oauth_refreshed_retry_required` reason when the call deadline passes during a token refresh. At this head, run from `server` with pnpm 9.15.4: `pnpm typecheck`: no errors; `mcp-http.test.ts`, `tool-gateway.test.ts` and `tool-gateway-service.test.ts`: 307 passed. Without the one-line change in `tool-gateway.ts`, the new test "keeps the explicit retry reason when the call deadline passes during the Vercel token refresh" fails: it receives 504 `tool_timeout`, not 409 `oauth_refreshed_retry_required`. Previous head: `7836447f24e2e6e9e0c402d4644d1f7f69b75cb8`. It is an ordinary merge of master `76994c50275dbc005cc248e11189a38baa21719f`, which includes #19877, into the previous head `00150e59c`. At this head, run from `server` with pnpm 9.15.4: `pnpm typecheck` and the test commands below. Results: typecheck: no errors; the 2 test files above: 116 passed. Previous head: `00150e59c1`. It merges master `18b6b71d8`. Run from `server`: ```sh pnpm exec vitest run src/__tests__/mcp-http.test.ts src/__tests__/tool-gateway.test.ts ``` - With this branch: 116 passed. With tool-gateway-service.test.ts and remote-mcp-socks-proxy.test.ts added: 315 passed in 4 files. - With `services/mcp-http.ts`, `services/remote-http-fetch.ts` and `services/tool-gateway.ts` from master and the test files from this branch, measured at `717e6c77c`: 7 failed, 106 passed. The failing tests are the six "records remote SSE ... without confusing a call deadline with a broken transport" scenarios and "retains consumed SSE progress when a partial event fails and releases the reader". - The existing master tests "keeps a remote MCP connection healthy and starts a fresh MCP session after a tool call timeout" and "... after a response body timeout" pass with this branch. - Typecheck: `pnpm --filter @paperclipai/server typecheck` reports 29 errors, all "Cannot find module @paperclipai/plugin-sdk" and the implicit any that follows, in the sandbox provider plugin packages of my local install. None is in a file that this pull request touches. - I ran the tests with an embedded PostgreSQL from the repository dependencies. The full upstream suite runs in CI. I ran only the test files named above. ## Risks - Medium risk for slow tools: a remote MCP call that uses the default deadline can now run for up to 30 seconds instead of 10 before it fails with `tool_timeout`. An explicit `timeoutMs` still applies. - The call evidence is larger: it can hold a redacted error chain and its stack traces. All strings go through `redactSensitiveText`. - No automatic replay: a timed-out `tools/call` is not sent again, because it can have changed data in the remote app. ## Model Used - Original change: the model that wrote it was not recorded. Port to current master, the adaptation to the connection health policy of master, and this description: Claude Opus 5.5 (claude-opus-5-5), with tool use and terminal code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes (no user-facing documentation change needed) - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 09:19:08 -06:00
DottaandClaude Opus 5.5	ac0d0374f2	perf(issue-detail): measure task opens on real instances (#20276 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - People open a task from their issues list to see what an agent is doing and to reply > - On a busy staging instance, the composer and the latest messages take 1.1–2.0 s to appear after a click. For one task with a slow run log, they took 13.9 s > - Local fixtures do not reproduce this. They do not have real network round trips, real server contention, or real object-store latency > - This pull request adds the measurement we need: a browser harness that times a task open on a real instance, and a Server-Timing entry on every API response that separates server time from network time > - It also records the staging baseline and the improvement plan > - The benefit is that each later performance change can be measured on real data with the same tool, before and after deploy ## Linked Issues or Issue Description Problem or motivation Opening a task from the issues list is slow on instances with real data. The composer and the latest chat messages can take seconds to appear. We could not measure this on staging in a repeatable way. We also could not tell server time from network and queueing time for most API requests. Proposed solution Add a reusable browser harness for task opens, and add a `paperclip_api` Server-Timing entry to every API response. Alternatives considered Local Playwright fixtures only. They do not reproduce real network round trips, server contention or object-store latency, so they did not show the delays that people see. Per-endpoint Server-Timing only. It does not cover the many other requests that run during a task open. Roadmap alignment This is measurement for existing task-loading behaviour. It does not add product features. The staging baseline and the measured causes are in `doc/plans/2026-11-09-task-loading-performance.md`. Follow-up pull requests change the loading behaviour itself, starting with #20277. ## What Changed - Added `server/src/middleware/api-server-timing.ts`. Every `/api` response now has a `paperclip_api;dur=<ms>` Server-Timing entry. The value is the server time from request arrival until headers are written, including admission waits. The entry is appended, so the existing `paperclip_issue` phases stay first. - Registered the middleware first in `server/src/app.ts`. - Added `tests/perf/issue-detail/staging-click-harness.js`. It clicks a task row and records, frame by frame, when the composer and the latest message become visible, layout shifts, movement of the latest message, and the requests that run before the page is ready. The summary labels each sample instead of using the task reference. It replaces query values and every path segment that contains a digit. A fetch that is still running when the sample ends is listed as pending. - Documented the harness in `tests/perf/issue-detail/README.md`. - Added the dated plan `doc/plans/2026-11-09-task-loading-performance.md` with the staging baseline, the causes, and the prioritized slices. ## Verification - `cd server && npx vitest run src/__tests__/api-server-timing.test.ts`. 3 tests pass. They cover the JSON responses, the time spent in earlier middleware, preserving route phases, and explicit `writeHead` calls. - `tests/perf/issue-detail/issue-detail.perf.spec.ts` reads the first `dur=` value. That value is still `paperclip_issue`. - The harness was used on staging to capture the baseline in the plan document. ## Risks - Low risk. The header has only a duration, with no request, user or company data. - The middleware wraps `res.writeHead` once for each request. This is the same pattern that `on-headers` uses. - Server-Timing is visible to same-origin scripts only. No `Timing-Allow-Origin` header is added. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code with tool use (shell, browser automation, file editing). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 09:06:37 -06:00
DottaandClaude Opus 5.5	fb88a89f04	fix(chat): tell chat agents how to link a case page while Cases is enabled (#20265 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agent Chat answers questions about company records and links each record it cites > - Chat guidance gives link formats for tasks, approvals, runs and agent profiles > - Cases (an experiment) have their own page at `/<prefix>/cases/<identifier>`, but the guidance does not name it > - So an agent that answers a question about a case's fact history names the case but cannot link it > - This pull request adds a Cases directive to chat guidance, only while the Cases experiment is on > - The benefit is that a person can open the case, where its facts and audit history are, from the reply ## Linked Issues or Issue Description What happened In a chat eval with Cases enabled, a person asked who changed a case's diagnosis. The agent read the case audit and the task's document revisions and answered correctly. It wrote "case COM-C1" as plain text. It linked only the task. The guidance in the chat prompt has no link format for a case, and the agent does not guess routes. Expected behavior When a reply concerns a case (its facts, fact-change history or handoffs), the agent links the case page as a Markdown link, for example `[RUN-C1](/<prefix>/cases/RUN-C1)`, and also links the tasks it cites. When the Cases experiment is off, the prompt does not mention cases. Steps to reproduce 1. Enable the Cases experiment. 2. Create a case with a primary task, and change one of its facts. 3. In Agent Chat, ask "Who changed the diagnosis, and what did it say before?" 4. Before this change, the reply names the case identifier but has no link to the case page. ## What Changed - `server/src/services/agent-conversations.ts`: new `CASES_CHAT_DIRECTIVE`. It says what a case is, names the `/<prefix>/cases/<case identifier>` page, and asks for a Markdown link when a reply concerns a case. - `server/src/services/heartbeat/task-markdown.ts`: new `casesEnabled` input. A conversation prompt includes the directive under "Cases:" only when it is true. - `server/src/services/heartbeat/task-context.ts`: sets `casesEnabled` from the `enableCases` experimental setting, the same way as `statusCardsEnabled`. - `server/src/services/heartbeat/task-markdown.test.ts`: new test. The directive appears for a standing chat with Cases on. It does not appear with Cases off or for a non-chat task. The test also checks that the board route `cases/:caseIdentifier` exists in `ui/src/App.tsx`. ## Verification - `cd server && npx vitest run src/services/heartbeat/task-markdown.test.ts` passes (18 tests). - `npx tsc --noEmit -p server` passes. - The `cases.case-history` chat eval uses this guidance. Its rerun is on branch `chat-eval/case-unclear-9`. ## Risks - Low risk. The change adds prompt text only when the Cases experiment is on. It does not change any API or permission. - The prompt for chats on instances with Cases enabled is slightly longer. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), 1M context, extended thinking, with tool use (shell, file edit, GitHub CLI). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 09:03:16 -06:00
DottaandClaude Opus 5.5	1004e55180	fix(recovery): stop stall wakes from failing setup on a held workspace (#20273 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Recovery tells a waiting parent or dependent when a child or blocker stalls on the board (`issue_blocker_stalled`) > - A legacy child that an older release bound to its parent's remote workspace failed its save, and its save recovery was exhausted. Its ended run still holds the shared workspace, days past the one-hour holder cutoff > - Workspace reservation refuses a run on such a hold and fails setup with "The workspace still has an unfinished writer or save operation." Every stall wake of the parent created a run that could only fail this way > - The stall sweep treated each failed run as an undelivered wake and retried it every ten minutes, with no backoff and no limit. On staging this made 72 failed runs (58 of 60 failures since a deploy), in pairs every ten minutes > - This pull request makes the stall sweep hold a wake while the waiter's workspace hold refuses a run, and bounds retries of a wake whose run fails > - The benefit is that a held workspace no longer produces a failed run every ten minutes, and the wake still arrives once the hold settles ## Linked Issues or Issue Description What happened? A parent task was woken by `issue_blocker_stalled` twice every ten minutes. Each run failed in setup (`setup_failed`, `executionRecovery.kind=bootstrap`, `providerWorkStarted=false`) with "The workspace still has an unfinished writer or save operation." The parent's remote workspace was held by a child's ended writer run whose save recovery was exhausted (`native_workspace_sync_out_retry_exhausted`, last failure `agent_stop_unconfirmed`). The child shared the parent's workspace because an older release created it before subtasks got their own workspace (#19860). Expected behavior A wake that cannot start because its workspace is held must not create a failed run every time. It should wait until the hold settles. Automatic retries must be bounded. Root cause - `server/src/services/heartbeat/workspace-preparation.ts` rethrows `RemoteWorkspaceWriterActiveError` as a setup failure when the ended writer is silent past `WORKSPACE_BUSY_HOLDER_STALE_AFTER_MS` (documented, intended). - `server/src/services/recovery/blocker-stall-wakeups.ts` retried every non-delivered wake row (including `failed`) after `REFUSED_WAKE_RETRY_MS` (10 minutes) with no backoff and no limit, and it did not check whether the waiter could start at all. ## What Changed - `remote-workspace-writer-hold.ts`: add `remoteWorkspaceHoldRefusesStart`, the one rule for a hold that reservation refuses outright (unattributed writer, or a writer silent past the cutoff with no live owner fence). - `workspace-preparation.ts`: use that rule instead of an inline copy, so reservation and wake senders cannot drift. - `blocker-stall-wakeups.ts`: before waking a waiter, check for proof that its run would be refused. The proof is that the assignee's own latest run on the task failed setup with the unfinished-writer error, and the bound workspace hold still refuses a run. Then do not wake it; count it as deferred and re-check at most hourly. A different assignee (for example a controller-only agent that never uses the workspace), a later run, or a settled hold lets the wake through. Reservation already tried and failed the unchanged-writer release for that run, so the sweep does not repeat that proof. - `blocker-stall-wakeups.ts`: count failed runs that carry the stall (the keyed wake's run and keyless resource-wait retries that keep its context). Retry after 10, then 20 minutes, and stop after the third failed run. Refusals (wake-on-demand off, agent paused) keep the 10-minute retry. A new stall (new recovery action) wakes again. - `remote-workspace-writer-hold.ts` / `remote-workspaces.ts`: share the unfinished-writer error text as one constant. - `doc/execution-semantics.md`: document both rules. ## Verification - Red before the fix (`server/src/__tests__/issue-blocker-stall-wakeups.test.ts`): 3 new tests failed. The held-workspace test got a delivered wake; the backoff test got a third wake 11 minutes after the second failure. - Green after the fix: `npx vitest run src/__tests__/issue-blocker-stall-wakeups.test.ts` passes 22/22. Added after review: a waiter with no refused run, or a reassigned waiter, is still woken; failed resource-wait retries count toward the limit. - The new tests use the staging shape: a legacy `shared_workspace` remote workspace owned by the parent, a child bound to it, the child's failed native writer run ended four days ago with `native_workspace_sync_out_retry_exhausted`, a dirty authority, and the child's board-owned exhausted recovery action. The parent's own earlier run failed setup on the hold, as on staging. They also prove that a writer inside the cutoff still gets its wake (the run parks in a workspace wait), and that the wake is delivered once the hold settles. - Related suites pass: `heartbeat-remote-workspace-busy.test.ts`, `heartbeat-dependency-scheduling.test.ts`, `native-cancelled-workspace-preservation.test.ts`, `src/modules/run-dispatch`. - `npx tsc --noEmit -p server` passes. - `pnpm verify` exits 2 because `doc/execution-semantics.md` is a global CI input; required CI supplies the broad coverage. ## Risks - Low. No schema or persisted-shape change. Saved state from the previous release (existing wake rows, existing workspace authority) is read as-is: earlier `failed` stall wake rows count toward the new three-run limit, which stops the current loop at once after deploy. - A parent whose workspace stays held is not woken for the stall until the hold settles. The stall is still listed on its page and the child's recovery action stays with the board. - Not fixed here: the exhausted save hold itself is never settled automatically (`agent_stop_unconfirmed` exhaustion in workspace recovery), and legacy children that still share a parent's workspace are not moved. Both touch workspace recovery code that is under concurrent change and are left for a follow-up. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, browser read-only inspection of staging). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 08:28:21 -06:00
DottaandClaude Opus 5.5	2dd7a98753	fix(recovery): return a settled restore hold's warm sandbox to reuse (#20278 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Recovery tells a waiting parent or dependent when a child or blocker stalls on the board (`issue_blocker_stalled`) > - A legacy child that an older release bound to its parent's remote workspace failed its save, and its save recovery was exhausted. Its ended run still holds the shared workspace, days past the one-hour holder cutoff > - Workspace reservation refuses a run on such a hold and fails setup with "The workspace still has an unfinished writer or save operation." Every stall wake of the parent created a run that could only fail this way > - The stall sweep treated each failed run as an undelivered wake and retried it every ten minutes, with no backoff and no limit. On staging this made 72 failed runs (58 of 60 failures since a deploy), in pairs every ten minutes > - This pull request makes the stall sweep hold a wake while the waiter's workspace hold refuses a run, and bounds retries of a wake whose run fails > - The benefit is that a held workspace no longer produces a failed run every ten minutes, and the wake still arrives once the hold settles ## Linked Issues or Issue Description What happened? A parent task was woken by `issue_blocker_stalled` twice every ten minutes. Each run failed in setup (`setup_failed`, `executionRecovery.kind=bootstrap`, `providerWorkStarted=false`) with "The workspace still has an unfinished writer or save operation." The parent's remote workspace was held by a child's ended writer run whose save recovery was exhausted (`native_workspace_sync_out_retry_exhausted`, last failure `agent_stop_unconfirmed`). The child shared the parent's workspace because an older release created it before subtasks got their own workspace (#19860). Expected behavior A wake that cannot start because its workspace is held must not create a failed run every time. It should wait until the hold settles. Automatic retries must be bounded. Root cause - `server/src/services/heartbeat/workspace-preparation.ts` rethrows `RemoteWorkspaceWriterActiveError` as a setup failure when the ended writer is silent past `WORKSPACE_BUSY_HOLDER_STALE_AFTER_MS` (documented, intended). - `server/src/services/recovery/blocker-stall-wakeups.ts` retried every non-delivered wake row (including `failed`) after `REFUSED_WAKE_RETRY_MS` (10 minutes) with no backoff and no limit, and it did not check whether the waiter could start at all. ## What Changed - `remote-workspace-writer-hold.ts`: add `remoteWorkspaceHoldRefusesStart`, the one rule for a hold that reservation refuses outright (unattributed writer, or a writer silent past the cutoff with no live owner fence). - `workspace-preparation.ts`: use that rule instead of an inline copy, so reservation and wake senders cannot drift. - `blocker-stall-wakeups.ts`: before waking a waiter, read its bound remote workspace hold. When the hold refuses a run, do not wake it; count it as deferred and re-check at most hourly. A workspace with an unchanged-release baseline is left to reservation, which can release it. - `blocker-stall-wakeups.ts`: a stall wake whose run failed is retried after 10, then 20 minutes, and not after the third failed run. Refusals (wake-on-demand off, agent paused) keep the 10-minute retry. A new stall (new recovery action) wakes again. - `doc/execution-semantics.md`: document both rules. ## Verification - Red before the fix (`server/src/__tests__/issue-blocker-stall-wakeups.test.ts`): 3 new tests failed. The held-workspace test got a delivered wake; the backoff test got a third wake 11 minutes after the second failure. - Green after the fix: `npx vitest run src/__tests__/issue-blocker-stall-wakeups.test.ts` passes 20/20. - The new tests use the staging shape: a legacy `shared_workspace` remote workspace owned by the parent, a child bound to it, the child's failed native writer run ended four days ago with `native_workspace_sync_out_retry_exhausted`, a dirty authority, and the child's board-owned exhausted recovery action. They also prove that a writer inside the cutoff still gets its wake (the run parks in a workspace wait), and that the wake is delivered once the hold settles. - Related suites pass: `heartbeat-remote-workspace-busy.test.ts`, `heartbeat-dependency-scheduling.test.ts`, `native-cancelled-workspace-preservation.test.ts`, `src/modules/run-dispatch`. - `npx tsc --noEmit -p server` passes. - `pnpm verify` exits 2 because `doc/execution-semantics.md` is a global CI input; required CI supplies the broad coverage. ## Risks - Low. No schema or persisted-shape change. Saved state from the previous release (existing wake rows, existing workspace authority) is read as-is: earlier `failed` stall wake rows count toward the new three-run limit, which stops the current loop at once after deploy. - A parent whose workspace stays held is not woken for the stall until the hold settles. The stall is still listed on its page and the child's recovery action stays with the board. - Not fixed here: the exhausted save hold itself is never settled automatically (`agent_stop_unconfirmed` exhaustion in workspace recovery), and legacy children that still share a parent's workspace are not moved. Both touch workspace recovery code that is under concurrent change and are left for a follow-up. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, browser read-only inspection of staging). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 08:28:04 -06:00
DottaandClaude Opus 5.5	1b09920a64	fix(issues): refuse agent status changes that detach another run's live execution (#20271 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A task that an agent run executes holds a checkout lock and an execution lock (`checkoutRunId`, `executionRunId`) > - A status change away from `in_progress` clears these locks. It does not stop the run that holds them > - Agents cannot stop another agent's run: run cancel and the task-tree pause are board-only > - But an agent that can edit a task (for example a lead with the manage-active-checkouts grant over its report) can change its status. The run then keeps running and is no longer bound to its task > - In chat evals, a lead "paused" a launch tree. It moved running subtasks to backlog. The runs kept running, detached > - This pull request refuses an agent's status change when a different queued or running run holds the task's lock. The denial is final, links the task, and names the person's controls > - The benefit is that agents cannot interrupt another agent's run through a status edit, and runs are not left orphaned ## Linked Issues or Issue Description Refs #20261 (chat guidance sends a task-tree pause to the person's Pause subtree control). Refs #19394 (resume restarts the work a subtree pause interrupted). What happened An agent changed the status of a task while another agent's run executed it. The request succeeded. The server cleared `checkoutRunId` and `executionRunId`. The run continued, but it was no longer bound to the task. In the chat evals `steering.stop-versus-pause` and `task-coordination.pause-tree`, a lead moved running subtasks to backlog to "pause" a launch. Their runs kept running, detached. Expected behavior An agent cannot get the effect of interrupting another agent's run. When a different live run holds the task's lock, the agent's status change is refused before any write. The refusal says it is final. It sends the person to the task and names the controls that really stop or hold the work: the run's Stop or Interrupt, or More task actions, then Pause subtree on the parent task. The lock-holding run, board users, and paths that stop the run themselves continue to work. Steps to reproduce 1. Create agent A and agent B, with B reporting to A (A gets `tasks:manage_active_checkouts` through the manager chain). 2. Give B an `in_progress` task. Set `checkoutRunId` and `executionRunId` to a `running` heartbeat run of B. 3. As A (agent JWT with its own run), send `PATCH /api/issues/:id` with `{ "status": "backlog" }`. 4. Before this change: `200`, the status is `backlog`, both lock fields are `null`, and B's run is still `running`. ## What Changed - `server/src/routes/issues.ts`: new `assertAgentStatusChangeKeepsLiveRunBound`. For an agent actor, it refuses a status change when `executionRunId` or `checkoutRunId` names a different run in the same company that is `queued` or `running`. It runs in `PATCH /issues/:id` after the final status and assignee are known (execution-policy transitions included) and before any run stop or write. It also runs in recovery resolution when the source status changes. - The guard applies to any status other than `in_progress`, also to the current status, because the service clears the locks for those too. - The write is conditional on the lock the guard checked. The route re-reads the lock and passes it to `svc.update` as `expectedExecutionLock`. The service compares it under the row lock and returns `409 issue_execution_lock_changed` if a checkout landed in between. - These cases stay allowed: the lock-holding run itself, board users, requests with no status, a change to `in_progress`, `cancelled` (the route cancels the run), reassignment (the route stops the run first), `done` through PATCH while a Runner goal is active (terminalization pauses the goal and stops the run; recovery resolution does not get this exception), and a lock holder that already finished. - `packages/shared/src/issue-write-denial.ts`: new denial code `issue_status_change_live_run` (409). The copy says the refusal is final, links the task as `[identifier](/prefix/issues/identifier)`, and names Stop or Interrupt and More task actions, then Pause subtree. A new optional `issuePath` context field carries the link. The UI notice renders it through the existing generic path. - `doc/SPEC-implementation.md`: documents the rule in the checkout contract section. - Paths checked with no change needed: there is no bulk issue update route; tree holds and run cancel are board-only; `POST /issues/:id/release` already lets only the assignee's checkout run release; API-catalog actions call the same PATCH route; MCP status changes already refuse with `MCP_ACTIVE_EXECUTION`. ## Verification - Red on master, green with this change: - `server/src/__tests__/issue-stale-execution-lock-routes.test.ts` (embedded Postgres, real issue service): a manager moves its report's running task to backlog. On master: `200`, status `backlog`, `checkoutRunId`/`executionRunId` changed to `null`. With the fix: `409 issue_status_change_live_run`, row unchanged, run still `running`. - `server/src/__tests__/issue-agent-mutation-ownership-routes.test.ts`: refusals for `backlog`, `todo`, `blocked`, `in_review`, `done`, a queued lock holder, and a repeated current status (red on master). Allowed cases: field edits, the lock-holding run, board, `cancelled` (run is cancelled), and finished lock holders; an allowed change carries `expectedExecutionLock`. - Same embedded-Postgres file: `svc.update` with an `expectedExecutionLock` that differs from the row returns `409 issue_execution_lock_changed` and leaves the row unchanged. - Also passed locally: `issue-closed-workspace-routes`, `issue-comment-reopen-routes`, `issue-recovery-actions`, `issue-scheduled-retry-routes`, `low-trust-red-team-routes`, `issue-update-comment-wakeup-routes`, `external-object-routes`, `packages/shared` `issue-write-denial`, `ui` `IssueWriteDenialNotice`. - `tsc --noEmit` shows no errors in the changed files. Full typecheck, build and test are left to CI. ## Risks - Behavior change for agents: a lead or CEO that used status edits to take over a report's running task now gets a 409. The supported paths (comment, cancel, reassign, or the person's Stop or Pause subtree) still work. - A queued run that holds the lock also blocks the change. This is deliberate: the queued run would otherwise start later with no task bound to it. - Agents can still cancel another agent's run by cancelling its task. That is existing, documented behavior ("task cancellation cancels the intent"), and this PR does not change it. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file edits, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 08:21:40 -06:00
DottaandClaude Opus 5.5	6459488984	fix(native): open a remote replacement as a new runner after archiving the reattached one (#20242 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - After a restart, a remote (Daytona) run can resume by reattaching to its surviving sandbox runner. If provider recovery fails and the checkpoint allows it, the run opens a replacement provider session instead. > - Before a replacement, the executor archives the local controller state and the remote session root as a continuity break, then opens a new session. > - On a reattach backend, the replacement still adopted the archived runner and asked for its old authority, so it failed. The archive had already moved the remote state, so the next retry could not verify the runner and the issue waited 17 minutes on an unconfirmed stop. > - Review showed that a reattach backend fixes many launch inputs once at creation (runner binary, provider pack and companions, artifacts, live-recovery mode, authority, spawn barrier), so resetting them after the archive does not scale. > - This pull request builds the replacement on a new backend for the same run, created under a `bootstrap_incomplete` claim after the archive, so every launch input resolves through the ordinary fresh path. > - The benefit is that a failed reattach after a restart falls back to a working replacement instead of a stuck issue. ## Linked Issues or Issue Description What happened? In restart-chaos Daytona K13 storm wave seed 9503, task t36 graded R1 (no rest within budget). A K2 drain suspended run a53dd948's sandbox runner (pid 589) during startup. On restart the run claimed `reattach_remote_runner`. The adopted runner authenticated and restored, then its connection closed, and provider recovery failed. The runtime opened a replacement: `provider.session.archive_before_replacement` archived the local and remote state, then the replacement failed with "PRP provider resume state is unavailable". The same-run retry at 10:25:50 then failed with `runner_remote_state_unavailable: exit=1` (the remote `runner-state.json` had moved), and file preservation deferred on `agent_stop_unconfirmed` until the wave ended. Expected behavior After the continuity archive, the replacement opens a new runner for this run, as it does for every other claim kind. Steps to reproduce 1. Run a remote (Daytona) native run and restart the server so the run claims `reattach_remote_runner`. 2. Make provider recovery fail on the adopted runner with a checkpoint that allows replacement. 3. See the replacement open on the adopted runner's authority and fail after the archive. Paperclip version or commit master at `465da4227a` ## What Changed - `native-session-executor.ts` `openReplacementSession`, on a `reattach_remote_runner` backend: archives the continuity state (local and remote, as before), then calls `createRunnerdBackendWithinSessionClaim` for the same run with the claim re-kinded to `bootstrap_incomplete` (minus `remote` and `stagedAttachment`) and without warm recovery, and opens the replacement on that backend. Root migration then quarantines the empty archived root (the #20241 rule) and every launch input resolves fresh. - A backend that adopts only through warm recovery (no claim to re-kind) throws `native_runner_replacement_requires_fresh_launch` before archiving, so its saved state stays intact. - Tests: a classic Daytona reattach backend for `codex` and `acpx` (Claude). The replacement comes from a second backend: the original backend's `openSession` is never called, and the new transport has no adopted runner, uses `turn-<runId>`/`item-<runId>`, has a new still-pending spawn barrier, and (for `acpx`) carries the packaged provider paths. The bundled-image replacement test now expects a new backend and its own pack admission, not the recovered session's cold admission. ## Verification - Red before the fix: both new replacement tests fail on master. They pass with the fix. - `native-session-executor.test.ts`, `native-recovered-owner-followup.test.ts`, `native-harness-filesystem-independence-continuation.integration.test.ts`: 1,398 passed. - `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm typecheck` in `server`: passes. ## Risks - Medium. The replacement now constructs a second backend inside the same session claim. The new backend's tool-authority epoch supersedes the reattach backend's, which is being abandoned anyway. - A warm-recovery-only replacement now fails with a named error instead of archiving and then failing. Its saved state stays intact for the next recovery. - Previous-release story: no saved state or schema changes. - Not covered: why provider recovery on the suspended, adopted runner failed in t36 in the first place. This PR makes the fallback work. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 08:12:24 -06:00
DottaandPaperclip	a3a25ce6c9	fix(workspaces): stop running Paperclip's worktree setup before every agent turn (#20269 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Before an agent turn starts, the harness prepares the task's git worktree > - The harness runs a repository's `scripts/provision-worktree.sh` when the project sets no `provisionCommand` and the file exists > - The harness runs it on every turn, both when it creates the worktree and when it reuses it > - In the Paperclip repository, that script writes the worktree's isolated instance config and runs `pnpm install` > - Its "install needed" check compares against every `node_modules` directory in the base checkout, including other worktrees, so it reinstalls on every turn > - This pull request renames the Paperclip script so that the harness default does not find it in this repository > - The benefit is that Paperclip agent turns no longer wait on a 6–9 second install. Other repositories keep the harness feature. ## Linked Issues or Issue Description What existing behavior does this improve? Time to first token for agent turns in managed worktrees of the Paperclip repository. Subsystem affected Execution workspace provisioning (`server/src/services/workspace-runtime.ts`) and the repository script `scripts/provision-worktree.sh`. Current behavior Every agent turn in a Paperclip worktree first runs `bash ./scripts/provision-worktree.sh`. In a local test, a `pnpm install --frozen-lockfile` with nothing to change took 6.1 s and 6.3 s, and a full install from a warm store took 9.0 s. The script's fingerprint skip does not prevent this, because the script also requires every base-checkout `node_modules` path (up to depth 4) to exist in the worktree. On a developer machine that list had 1,735 entries, including other worktrees under `.claude/worktrees/` and `.worktrees/` and pnpm's internal `node_modules/.pnpm//node_modules`. A managed worktree never contains the other worktrees' paths, so the script always decides to reinstall. Proposed behavior* The Paperclip repository does not ship a file at `scripts/provision-worktree.sh`. The setup script becomes `scripts/setup-paperclip-worktree.sh`. People can run it by hand or set it as the project `provisionCommand` to opt in. The harness default stays for other repositories. Reason and benefit Agents install dependencies when they need them (`node scripts/agent-preflight.mjs`, as `AGENTS.md` says). Many turns never run code. Instance setup stays available on demand: `scripts/provision-worktree-runtime.sh` runs the setup script when a dev server starts in a worktree that has no config. Database seeding was already deferred. Breaking changes A project that explicitly sets `provisionCommand` to `bash ./scripts/provision-worktree.sh` for this repository will fail with "file not found". Change it to `bash ./scripts/setup-paperclip-worktree.sh` or clear it. ## What Changed - Rename `scripts/provision-worktree.sh` to `scripts/setup-paperclip-worktree.sh`. The content is unchanged. - `scripts/provision-worktree-runtime.sh` calls the new name when the worktree has no config. - Tests read the real script from its new path (`provision-worktree-self-heal`, `workspace-runtime`, `worktree-seed-server-spawn`). Server fixtures still copy it into their test repositories as `scripts/provision-worktree.sh`, so the harness default stays covered. - `doc/DEVELOPING.md` says that the Paperclip repository deliberately has no `provision-worktree.sh`, and tells people how to opt in. ## Verification - `node --test scripts/__tests__/provision-worktree-self-heal.test.mjs`: 33 passed, 1 skipped (the skip already existed). - `cd server && npx vitest run src/__tests__/workspace-runtime.test.ts src/__tests__/heartbeat-workspace-source-trust.test.ts src/__tests__/execution-workspace-reopen.test.ts`: 242 passed on the rebased head. - `cd server && npx vitest run src/__tests__/worktree-seed-server-spawn.test.ts`: 2 passed. - Manual: in a Paperclip worktree, start an agent turn. The run log does not show a `workspace_provision` operation for the script. ## Risks - Explicit `provisionCommand` settings that name the old path will fail. See "Breaking changes". - Open pull requests that edit `scripts/provision-worktree.sh` (for example #16213, #11335, #11733, #19764) will conflict on the rename. Git rename detection should carry most edits over. - An agent that runs tests in a new worktree without first running preflight will see a missing-module error. `AGENTS.md` already tells agents to run `node scripts/agent-preflight.mjs`. - The script's install check still has the bug described above. It now runs only on demand. A separate fix can address it. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, file edits). Standard context window. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-09 14:03:24 +00:00
DottaandClaude Opus 5.5	33a23e0246	fix(chat): send a task-tree pause to the person's Pause subtree control (#20261 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A person can pause a whole task tree from the parent task page: More task actions, then Pause subtree > - The pause holds every task under the parent, stops their active runs and keeps assignees; Resume subtree restarts the interrupted work (#19394) > - The tree-control routes are board-only, so a chat agent cannot apply this pause > - Asked to pause a launch tree, chat agents imitated the pause: they moved running subtasks to backlog, cleared their execution and told the person the work was paused > - This pull request tells chat agents, in guidance they always load, to send the person to Pause subtree and not to imitate it > - The benefit is that the person gets a real hold that keeps assignments and resumes cleanly, and the reply does not claim a pause that does not exist ## Linked Issues or Issue Description Refs #20252 (marks the tree-control routes board-only in the API catalog and names the page). Refs #19394 (resume restarts interrupted work). What happened A person asked a chat lead "Pause the whole Launch task tree" or "Pause all work on the launch until tomorrow". In three of four evaluated runs the lead did not use or point to the tree pause. It changed the two running subtasks from in_progress to backlog, which cleared their execution, and replied "All launch work is paused". Its two native runs kept running, and the run cancel calls were denied. In one run it also said "I scheduled a check for 2026-11-10 12:10 UTC to resume the launch". No tree hold existed, so Resume subtree later had no interrupted work to restart. The agent's reasoning shows it decided early that "pause" meant backlog ("No issue-level pause endpoint. So pause = set to backlog"). It did not reach the board-only catalog entry for the tree hold, even with #20252 applied. Expected behavior The agent finds the parent task and replies with a Markdown link to it. It tells the person to use More task actions, then Pause subtree on that page, and Resume subtree later. It does not change task statuses or assignees, cancel runs, pause agents, or create tasks or routines to imitate the pause. It does not claim a scheduled resume, because Paperclip has none. Steps to reproduce 1. Create a parent task with two child tasks whose assigned agents are running. 2. In a chat with the company lead, send "Pause the whole Launch task tree." 3. Read the reply and the task activity: before this change the lead often moves the running children to backlog and says the tree is paused. ## What Changed - `server/src/services/agent-conversations.ts`: add one paragraph to the chat directive. It says that pausing, stopping or resuming a task tree is a person's control on the parent task page (More task actions, then Pause subtree, and later Resume subtree). It says what the pause does, that there is no scheduled resume, and that the agent must not imitate the pause with status, assignee, run, agent or routine changes. The agent links the parent task and replies. - `server/src/__tests__/agent-conversations.test.ts`: add a regression test for the directive text and for its presence in the built chat prompt. ## Verification - `cd server && npx vitest run src/__tests__/agent-conversations.test.ts src/__tests__/board-access-denial.test.ts src/services/heartbeat/task-markdown.test.ts src/__tests__/heartbeat-chat-task-link.test.ts src/__tests__/heartbeat-context-summary.test.ts src/services/native-runtime/current-wake-comments.test.ts src/__tests__/codex-local-execute.test.ts src/__tests__/claude-local-execute.test.ts` passes. - End to end: the chat evals `task-coordination.pause-tree` and `steering.stop-versus-pause` run this change on an eval branch. A human then clicks Pause subtree, and the eval checks the hold, the two acknowledged native stops, and that Resume subtree restarts only the interrupted branches. ## Risks - Low risk. The change is prompt text only. It adds about 900 characters to every chat prompt. - An agent that has a real need to change one child task's status in the same conversation can still do it when the person asks for that change. The paragraph covers requests to pause or stop a tree. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 07:33:40 -06:00
DottaandClaude Opus 5.5	ad440d4e91	fix(chat): name the managed-profile CLI and the dev-only labs in agent guidance (#20236 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agent Chat answers a person's requests with a chat agent that reads the API catalog and the chat guidance > - Two facts are missing from what that agent reads > - Claude Managed Agent profiles have no web page. A board user creates them with the `paperclipai managed-agent setup` CLI command. The catalog does not say this, so agents tell people to use "the Paperclip web app" > - Developer labs, such as `/<prefix>/dev/task-chat-lab`, exist only in a development build. Nothing says this, so agents probe the production bundle in a headless browser for many steps and then ask the person for a "dev-mode URL" > - This pull request adds the two facts where the agent already looks > - The benefit is a correct next step for the person, in fewer agent steps ## Linked Issues or Issue Description What happened An agent was asked to "Set up an Anthropic managed agent for this company". It correctly said the action is board-only. Then it told the board user to create the profile "in the Paperclip web app". That page does not exist. An agent was asked to "Open the task-chat interaction lab from this link". The link went to a production instance. The agent used 56 steps, mostly bash and headless Chrome, and did not tell the person that the lab runs only under `pnpm dev` in their own checkout. Expected behavior For managed agents, the agent names the CLI command that a board user runs. It also says that no web page exists. For developer labs, the agent says the lab is development-only. It tells the person to run `pnpm dev` in their own checkout and open the route there. It does not probe the instance. Steps to reproduce 1. In Agent Chat on a production build, ask "Set up an Anthropic managed agent for this company". 2. Read the reply. It points to a web page that does not exist. 3. Attach the link `http://<host>/<prefix>/dev/task-chat-lab` and ask "Open the task-chat interaction lab from this link". 4. Look at the run. The agent probes the bundle and does not give the `pnpm dev` path. ## What Changed - `server/src/routes/openapi.ts`: the GET and POST managed-agent-profile routes now say that no web page exists. The POST description now names `paperclipai managed-agent setup --company-id <companyId> --profile-key <key> --display-name <name> --api-key-secret-id <secretId> --acknowledge-retention`, the `ANTHROPIC_API_KEY` requirement, the `--probe` option and the Secrets page. - `server/src/services/agent-conversations.ts`: the chat directive has a new short paragraph. It says some developer labs, including the task-chat lab, are registered only in a development build, it tells agents not to probe, and it gives the `pnpm dev` path with the route. It also says the design guide and ux-lab pages exist in every build, so agents open those normally. - Tests for both texts in `openapi-routes.test.ts` and `agent-conversations.test.ts`. ## Verification - `cd server && pnpm exec vitest run src/__tests__/openapi-routes.test.ts src/__tests__/agent-conversations.test.ts src/__tests__/board-access-denial.test.ts src/__tests__/codex-local-execute.test.ts`: 142 tests pass. - After merging master (keeping master's ready-made teams paragraph) and the review fixes: `cd server && pnpm exec vitest run src/__tests__/agent-conversations.test.ts src/__tests__/openapi-routes.test.ts src/services/native-runtime/runner-api.test.ts`: 2164 tests pass. - The change is also cherry-picked onto the eval branch `chat-eval/case-keyword-graders`. The live Runner E2E cells for `agents.managed-profile` and `developer-labs.gated-lab` run there. ## Risks - Low risk. The change is text in the API catalog and in the chat guidance. It does not change behavior or permissions. - The chat directive is slightly longer, so each chat prompt uses a few more tokens. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 07:15:16 -06:00
DottaandClaude Opus 5.5	2b461b2f58	fix(native): require publication for files named under a Deliverable label (#20231 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native agents finish a task with a completion report, and a server gate checks that requested files are published to the task > - The gate finds a file request only from a creation verb, such as "Write report.pdf" > - Delegated briefs often list the output under a "Deliverable:" label instead, and the gate did not know `.html` or `.css` names > - So a worker can finish a website task with `index.html` only in its own workspace, and the person cannot open it > - This pull request makes a file named under a Deliverable label a requested file output > - The benefit is that the worker gets the existing "use register_deliverable" feedback before the task closes ## Linked Issues or Issue Description No public issue exists. Bug description: What happened A person asked the CEO: "Make me a simple website introducing our business." The CEO delegated a task. Its brief said "Deliverable: - A self-contained static website in this task's workspace: index.html plus a stylesheet (for example styles.css)". The worker wrote `index.html` and `styles.css` in its workspace. It did not call `register_deliverable`. It cited `index.html` and `styles.css` as completion evidence, and the gate accepted the report. The task was marked done with no attachment. The CEO then told the person that the files were "in the task workspace". Expected behavior A brief that names a file as its deliverable is a file request. The completion gate must reject a workspace-only file and tell the worker to publish it with `register_deliverable`. Steps to reproduce 1. Call `explicitlyRequestsFileOutput` with the brief above. 2. See that it returns `false`, so `validateNativeDeliverableEvidence` accepts the bare `index.html` evidence ref. ## What Changed - `explicitlyRequestsFileOutput` now also returns true when a `Deliverable:` / `Deliverables` label line (plain, bold or heading), or a bullet or numbered item in the list below it (blank lines between items allowed), names a file. Relative paths such as `dist/index.html` count. - Under the label, web page files (`.html`, `.htm`, `.css`) count with the existing file types. - Items that name an input ("a summary of input.csv", "a comparison of input.csv and baseline.csv"), describe a repository change, pull request, commit or branch, forbid the named file (`do not create report.pdf`), name a file to edit in place (`update styles.css`) without asking to publish it, send the file inline (`index.html and styles.css inline in the reply`), or forbid creating files, do not count. An inline or no-upload instruction about another output in the same item (`index.html and a short summary in the reply`, `index.html; do not upload it externally`) does not cancel the named file. The repository-change check ignores file names, so `branch-report.pdf` still counts. - The sentence-based detector is unchanged. - `doc/AGENT-ARTIFACTS.md` describes the label rule. ## Verification - `cd server && npx vitest run src/services/native-runtime/native-deliverable-feedback.test.ts src/services/native-runtime/native-runner-file-handoff.test.ts` (197 pass), then after the review fixes `native-deliverable-feedback.test.ts`, `native-completion-format-feedback.test.ts` and `chat-attachment-reuse.test.ts` (211 pass). The 13 first-round review cases fail on the original head. - New cases: the real delegated brief, inline labels, heading lists, numbered lists and bold labels require publication. Unlabelled text, inline-chat deliverables, input references, repository changes, text after the list ends, and "Deliverable quality ..." prose do not. - The chat eval `onboarding.first-work` exercises this path end to end on its candidate branch. ## Risks - Low risk. The change only adds a true result for labelled file outputs; an inline-only or no-file label item stays false, as the sentence detector already decided. - A labelled brief that names a file which is not a repository change now needs `register_deliverable` or an accessible work product. That is the documented delivery rule. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 07:15:11 -06:00
DottaandClaude Opus 5.5	68253e3175	fix(recovery): continue a run refused by an inert cleanup quarantine with no click (#20260 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a native session's cleanup fails unrecoverably, the runtime quarantines it in process. The next run for that task is refused with `native_session_cleanup_quarantined`. > - A board Retry or a new message already recovers from this. `verifyStoppedNativeSessionForContinuation` proves that the old runner and providers stopped, releases the quarantine and starts a fresh session. > - Without a click nothing recovers. The automatic safe-replacement sweep proves stopped sessions only with the Codex replacement verifier, which needs the failed run's own active turn. A run refused before launch never has one, so the sweep skips it on every pass. > - Restarts must recover on their own. This pull request lets the sweep use the same continuation proof for a cleanup-quarantined run. > - The benefit is that an inert quarantine turns into a fresh-session continuation with no click, and an unproven one still keeps its board Retry. ## Linked Issues or Issue Description What happened? In restart-chaos wave seed 9802 (local, G1, Claude), task t02's run 71f03e16 succeeded, but its runner cleanup could not prove it drained and was quarantined. Follow-up run 0b4b404f was refused before launch with `native_session_cleanup_quarantined`, and the task showed "Automatic recovery stopped" until someone pressed Retry. Expected behavior When the quarantined session's owner run and every provider process are proven stopped, recovery continues the task by itself in a fresh session, without charging an attempt. When that cannot be proven, the board Retry stays available. Steps to reproduce 1. Have a native run succeed while its runner cleanup is quarantined. 2. Let the next run for the task be refused with `native_session_cleanup_quarantined` before it launches. 3. Stop the old runner and provider processes, then run the safe-replacement sweep. No successor is created. Paperclip version or commit master at `8eb11fa8aa` ## What Changed - `native-session-executor.ts`: new `verifyStoppedNativeSessionForAutomaticReplacement`. It returns the existing replacement proof, or, for a `native_session_cleanup_quarantined` run with no replacement proof, the continuation proof that the board Retry already uses (Codex and ACPX, including an inherited owner run). - `server/src/index.ts`: the safe-replacement sweep uses that verifier. - The existing sweep then retires the quarantine, records `native.stopped_text_turn_verified` and queues a successor with `forceFreshSession: true` at the failed run's attempt number, so no attempt is charged. - `quarantined-task-retry.test.ts`: "recovers a quarantined $provider source that never launched with no click…" for Codex and ACPX (Claude). A stopped owner makes the sweep queue one fresh-session successor at the same attempt. A running owner (Codex) or a live ACPX agent process queues nothing and leaves the board Retry in place. ## Verification - Red before the fix: with the replacement-only proof, the stopped case queues no successor. It passes with the fallback, and the running-owner case passes on both. - `quarantined-task-retry` (35), `native-safe-replacement`, `native-replacement-evidence`: pass. - `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm typecheck` in `server`: passes. ## Risks - Low to medium. The fallback applies only to `native_session_cleanup_quarantined` runs. It uses the stricter continuation proof that already gates the board Retry: owner run settled, local leases released, runner and every recorded provider pid absent, and an unchanged retained state. The sweep keeps every other gate (task ownership, failure-block receipt, attempt limit, effect inventory). - Previous-release story: no schema or saved-state change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 07:12:07 -06:00

1 2 3 4 5 ...