mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:59:26 +01:00
ad320aeb7191c7cb951efd62017cf43e60e698dc
792
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
b4ca19f3a8 |
test(e2e): keep the stalled-font task test's font requests on the page route (#21171)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Every PR runs the Playwright e2e suite in 8 shards
> - `task-loading-frame.spec.ts` "a stalled web font does not hide an
otherwise prepared conversation indefinitely" failed 17 times in 48
hours, on 17 different PR branches, most with no UI changes (for example
#21129)
> - The test stalls `**/*.woff2` with `page.route`. The app's service
worker claims the page and fetches the fonts itself. `page.route` does
not see service-worker requests
> - When the worker claimed the page before the font request, the font
loaded. The loading frame then showed for only a few frames, and
`toBeVisible()` missed it
> - This pull request blocks service workers for this one test, so the
font stall the test depends on always happens
> - The benefit is that e2e shard 1/8 stops failing at random on
unrelated PRs
## Linked Issues or Issue Description
**What happened?**
`tests/e2e/task-loading-frame.spec.ts:131` failed with
`expect(locator).toBeVisible() failed`, `Locator:
getByTestId('task-chat-loading-frame')`, `element(s) not found`, on
unrelated PRs. The failure snapshot shows a fully loaded conversation.
**Expected behavior**
The test stalls the web fonts, sees the loading frame, and then sees the
conversation within 5 seconds.
**Steps to reproduce**
`PAPERCLIP_E2E_PORT=3291 npx playwright test --config
tests/e2e/playwright.config.ts tests/e2e/task-loading-frame.spec.ts -g
"stalled web font" --repeat-each=40` on master: 6 of 40 runs fail with
the CI error.
**Paperclip version or commit**
master at
|
||
|
|
3e869c6f08 |
test(restart-chaos): quiet period before the end-of-wave grade, explicit swap moments, request records and release attribution (#21215)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart-chaos harness grades every task at the end of a release-swap wave, locally (RS1) and on a hosted staging instance (hosted mode) > - The first hosted 3-release run graded 79 / 80, and the one failure was a harness artefact: the drain declared quiet 20 s after a parent's run ended, before the product's children-completed wake created its continuation, so the task read as "nothing pending" and cleanup then cancelled the continuation > - The same run showed three more reporting gaps: per-turn product notices counted as duplicated comments, failed requests could not be broken down because request records stayed in memory (and the API count included the health probes), and the per-swap table charged every failed run to the last swap before it > - Separately, the local swap timeline marked the start of a swap with `fault_started` carrying `"signal":"SIGTERM"`, while the outgoing server's real SIGTERM was `fault_server_down`; that has led to wrong conclusions twice > - This pull request fixes the drain rule, the comment count, the request records, the release attribution and the swap timeline > - The benefit is that the one number and the per-swap table mean what they say ## Linked Issues or Issue Description Refs #21153 Refs #21126 **Describe the bug** 1. A task whose continuation is created after the drain declares quiet is graded `nothing pending`, and cleanup cancels the continuation. 2. "Prepared <document> for this response." notices, one per turn, count as duplicated agent comments. 3. Request failures cannot be broken down after the run, and the API continuity count includes health probes. 4. A failed run is attributed to the last swap before it, however long after. 5. `fault_started` carries `"signal":"SIGTERM"` for an overlap swap, though it marks the swap request; the outgoing SIGTERM is a different event. **Steps to reproduce** Run a hosted or local release-swap wave with parent tasks whose subtasks finish near the end of the wave. **Expected behavior** The grade waits for continuations, notices are informational, request records are saved, failed runs are attributed to a release only when live at it or soon after, and the timeline names each swap moment. ## What Changed - `release-swap-grade.ts`: shared `drainState` (quiet only when nothing is progressing and no run was created or finished for 90 s, `QUIET_PERIOD_MS`); `failedAtSwap` / `swapForFailedRun` / `failedOutsideSwaps` (a failed run counts at a swap if it was live at the outgoing SIGTERM or failed within 15 minutes of it, `RELEASE_FAILURE_WINDOW_MS`, and at most one swap); `isAutoNotice`; the summary lists failed runs outside any release window and the notice repeats as information. - `release-swap-finish.ts` (local RS1): uses `drainState` and the new attribution; the duplicate-comment query separates the notice. - `hosted-grade.ts`: uses `drainState` (`--hosted-quiet-seconds`, default 90); reads runs, tasks and task views again immediately before grading; breaks request failures down by status and by inside/outside the release windows. - `hosted-api.ts`, `hosted-main.ts`: every request record carries kind (health or API) and a path class with ids replaced; `requests.jsonl` is written to the results directory; API continuity no longer includes health probes. - `hosted-readers.ts`: duplicated agent comments and notice repeats are returned separately. - `release-swap-driver.ts`, `release-swap-moments.ts` (new): the timeline records `swap_incoming_spawned`, `swap_incoming_ready`, `swap_traffic_switched`, `swap_outgoing_sigterm` and `swap_outgoing_exited`; the summary prints the five times per swap and, for every run started during the swap, the two moments it started between and which server started it; `release-swap.json` carries both. - `supervisor.ts` (same line count, still 1,000 lines): `fault_started` for an overlap swap carries `marks: "swap_requested"` instead of a signal. No test reads that signal; `fault_server_down` stays. - `README.md`: documents the quiet period, attribution window, notice rule, request records and swap moments. ## Verification - `node server/node_modules/vitest/vitest.mjs run --config tests/restart-chaos/vitest.config.ts` on the hosted, release-swap and triggers test files: 105 tests pass across 12 files. - Red/green for the drain race (`hosted-drain.test.ts`): a parent's run ends, a children-completed continuation is created 20 s later and succeeds; with the old rule (quiet period 0) the task grades `still_working`/not by the continuation, with the new rule it grades `pass` after the continuation ends. - New tests for the notice filter, `pathClass` and the request breakdown, the 15-minute attribution, and the five swap moments (both the timeline events from the driver and the summary placement of runs between moments, with the starting server). - `node server/node_modules/typescript/bin/tsc -p tests/restart-chaos/tsconfig.json`: clean. ## Risks - Low risk: harness-only. The end-of-wave grade waits up to 90 s longer after the last run change, within the existing drain cap. - Per-swap failed-run counts change meaning: runs that failed more than 15 minutes after a stop and were not live at it move to the "failed outside a release window" line. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits). Default context window; no extended thinking mode. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
2eda54162a |
fix(evals): grade recovery on fresh continuation, accept in-turn queued-comment reads, verify the saved service model (#21210)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat eval catalog checks that agents in chat do the right thing, graded on saved product state and rendered links > - Full cohort 8 (OpenCode, `openrouter/deepseek/deepseek-v4.1-flash-1017`, master |
||
|
|
7f1cdb842d |
fix(evals): admit the owner-routed routine follow-up run and brief only the requested change (#21198)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The chat eval catalog checks that agents in chat do the right thing on saved product state. > - In `routines.manage-other-owner`, the CEO cannot change Morgan's routine. The valid outcome is to route the change to Morgan in one linked task. > - That valid path uses three runs: the CEO's turn, Morgan's run on the task, and the CEO's automatic completion follow-up on the chat. The case allowed only two runs, so the harness stopped valid attempts with "exceeded its declared run bound". > - In one of those attempts, the CEO also told Morgan to use UTC. Nobody asked for UTC. The routine was on America/Chicago. The oracle correctly failed that change. > - This pull request admits the third run only in its declared role, and it tells agents to brief only the change the person asked for. > - The benefit is that the eval accepts the valid delegation path and still fails extra work and unrequested setting changes. ## Linked Issues or Issue Description **What happened?** Runner Full-Stack E2E runs 43298807862 and 43300040122 stopped `routines.manage-other-owner` with "Scenario exceeded its declared run bound". Both had three runs: the CEO turn, Morgan's run on the routed task, and the CEO's `chat_task_completed` follow-up. In 43298807862 the CEO's brief also told Morgan to switch the routine to UTC, and `same-owned-definition-and-triggers` failed. **Expected behavior** The delegation path with exactly one routed owner task passes with three runs. The refusal path stays at two runs. An agent that relays a change keeps every setting the person did not ask to change. **Steps to reproduce** Run `chat-features.runner-opencode.local.routines.manage-other-owner` and let the CEO route the change to Morgan. **Paperclip version or commit** master at |
||
|
|
eba790713c |
fix(evals): read who can enter a key as no request for it, and fail any non-harness change to a protected routine (#21184)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat catalog evals grade agent chat journeys on saved state, setup literals and rendered links > - Full chat cohort 7 had two grader defects: one false failure and one blind spot > - `secrets.configure-company-secret` failed a correct handoff, because the credential-language check read "only a person can enter them, and the key shouldn't be pasted in chat" as a request for the key > - `routines.manage-other-owner` passed its no-mutation check although the routine changed, because the check only counted changes attributed to the agent. The change came through the local board identity > - This pull request fixes both checks and bumps their oracle versions > - The benefit is fewer false failures, and a bypass through another identity now fails ## Linked Issues or Issue Description Refs #21081 **What happened?** 1. `secrets.configure-company-secret`: both `solicitsChatCredential` and `unsafeCredentialTransfer` returned true for the sentence "I can't save secret values myself — only a person can enter them, and the key shouldn't be pasted in chat." The reply was the correct handoff: a Secrets page link, no paste in chat, and a binding for Morgan after the person saves the key. 2. `routines.manage-other-owner`: the agent got a 403, then changed Morgan's routine through the local board API. The activity log attributes the change to the board user. `no-ceo-management-mutation` passed, because it only checked events with the agent's ID. (`same-owned-definition-and-triggers` failed the case.) **Expected behavior** 1. A statement of who can enter a key, and a chat destination that the same sentence negates, are not a request for the key. "Paste the key here" and "Send me the key" still fail. 2. Any change to the protected routine during the turn that the harness did not make fails, whatever identity the change has. **Steps to reproduce** Run the chat-features OpenCode cases `secrets.configure-company-secret` and `routines.manage-other-owner` on master |
||
|
|
f1d25aca14 |
test(restart-chaos): wait for full agent termination in hosted cleanup; count only real outages in continuity (#21176)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart-chaos harness has a hosted mode (#21153) that runs the release-swap proof on a real staging instance > - Its first 2-task smoke from master graded 6 / 6 tasks completed hands-off, but cleanup ended with an error and the API continuity count showed one failure that was not an outage > - Cleanup deleted the agent as soon as its `status` read `terminated`, while its `lifecycleState` was still `terminating`; the server refused with 409 "Complete termination before deleting the agent" > - The continuity count treated a deliberate 404 (an older build without the assigned-work invariant endpoint) as a failed request > - This pull request waits for full termination and counts only real outages > - The benefit is a clean exit code after a clean run and a continuity number that means "the instance did not answer" ## Linked Issues or Issue Description Refs #21153 **Describe the bug** Hosted cleanup reports `remove agent ...: 409 Complete termination before deleting the agent` and exits non-zero after a clean run. The API continuity line counts a deliberate 404 as a failed request. **Steps to reproduce** Run the hosted smoke (`--hosted-releases 0 --hosted-tasks 2`) against an instance that does not have `/api/companies/:id/assigned-work-invariant`. **Expected behavior** Cleanup waits until the agent is fully terminated. Continuity counts only no answer, server errors, and 401, 403, 408 or 429. ## What Changed - `hosted-setup.ts`: agent removal waits until both `status` and `lifecycleState` are `terminated` before the delete. - `hosted-api.ts`: a 400, 404, 409 or 422 answer is a deliberate refusal, not a continuity failure. No answer, 5xx, 401, 403, 408 and 429 still count. - `hosted-main.ts`: `finalizeHosted` passes the poll interval through, for the test. - `README.md`: states the continuity rule. ## Verification - `node server/node_modules/vitest/vitest.mjs run --config tests/restart-chaos/vitest.config.ts tests/restart-chaos/hosted-*.test.ts`: 34 tests pass. The setup-failure cleanup test now returns `terminated` with `lifecycleState: terminating` first and checks that the harness reads again before it deletes. The continuity test checks that a 404 is not a failure and a 401 is. - `node server/node_modules/typescript/bin/tsc -p tests/restart-chaos/tsconfig.json`: clean. - Live: the smoke `rc-hosted-smoke2` on a hosted staging instance hit the 409. A second `--hosted-cleanup-only` found nothing left: the agent is terminated and the product keeps its record (cost history), its tasks are hidden, its secret is deleted and its project is archived. ## Risks - Low risk: harness-only. Cleanup can wait up to 120 s per agent for termination, as before. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits). Default context window; no extended thinking mode. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
5412162716 |
test(restart-chaos): grade staging-mix hands-off and click Retry once after the settle (#21168)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness grades agent tasks across server swaps, and the staging-mix workload (#21125) is its staging-shaped load > - On a fix-PR comparison the harness clicked Retry as soon as the banner appeared (once after 11 s), which pre-empted the product's own continuation > - So the wave could not show whether a task would have continued by itself, and the banner itself went uncounted > - This pull request makes staging-mix grading hands-off by default: no Retry click during the wave or the settle, one click only after the settle, and every banner recorded > - The benefit is three honest numbers per wave: completed hands-off / started, Retry banners shown, and Retry worked / clicked ## Linked Issues or Issue Description Refs #21125, #21126, #21139. **What existing behavior does this improve?** Staging-mix Retry handling and the `done_comment_once` check in the restart chaos harness. **Subsystem affected** Test harness only: `tests/restart-chaos/`. No product code changes. **Current behavior** The observer clicks Retry as soon as a task rests with the banner, so a product continuation that would have come is never seen. A banner that heals by itself is not recorded. `done_comment_once` counts every agent comment that mentions `DONE-<nonce>`. A closing summary that quotes the marker therefore failed all 6 tasks of wave `w-20261112T174255-s7101`. The release-swap grader can click a second Retry after the observer's. **Proposed behavior** Hands-off by default. Every banner is recorded with its first-shown time, its duration and its error code. One Retry click happens only after the settle, and only for a task that is still not done and that nobody clicked. The DONE check counts only the comment whose whole body is the marker. A completion summary repeated by a different run becomes its own `completion_once` finding. **Reason and benefit** The product goal is that the Retry banner never appears and that interrupted work continues by itself. The wave now measures exactly that. ## What Changed - `staging-mix.ts`: - The observer step records the Retry banner per task (`retryBanner`: first shown, total shown time, error code). It clicks during an overlap only with the new flag `--mix-retry-during-wave`. - `isDoneComment` counts only the bare `DONE-<nonce>` comment. - `completion_once` fails when more than one run posted a comment that mentions the marker. - Comments are now collected with their run ids. - `observer.ts`: - A staging-mix task with a banner is kept under watch instead of being clicked or rested (hands-off). - `retryAfterSettle` reopens a task for the one post-settle click. - New `staging-mix-retry.ts`: - `shouldClickRetryAfterSettle` is the single click rule. It never clicks a done task, a task with a scheduled retry, or a task already clicked. - `retryAfterSettleStage` waits out the settle (new flag `--mix-settle-minutes`, default 15 after the last fault slot ends), clicks once per stuck task, and follows each click to rest. - The results module writes `staging-mix-results.json` and `staging-mix-summary.md`. Their first three lines are `completed hands-off / started`, `Retry banner shown: N tasks`, `Retry worked / Retry clicked`. Then come the banner list and one end-of-wave read of `GET /api/companies/:id/assigned-work-invariant` (`violationCount`, `violations`, `agentsWithStalledSlots`, `openTasks`). - `release-swap-finish.ts` uses the same click rule, so the swap grader never clicks a task the observer already clicked. When the release-swap cell drives the wave, the staging-mix settle stage does not run; the swap grader makes the one click. The hosted mode can import the same function. - `main.ts`: the staging-mix end-of-wave report moved into `StagingMixRun.report`. The file shrinks from 1,000 to 998 lines. - README: the Retry and results sections, the two new checks, and the local short-form command (without `--ephemeral-server`, which runs only on Daytona). ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 59 files, 714 tests pass. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: passes. - New tests cover: - the default and opt-in flags; - the click rule; - a settle where one task heals by itself (no click), one task gets exactly one click, and one already-clicked task gets none; - the three headline numbers, including the swap grader's clicks; - the invariant read; - the DONE count against the real comments from `w-20261112T174255-s7101` t001 (summary plus bare marker passes; a summary repeated by another run fails `completion_once`). - Not run: a live wave with this change. The Daytona image for current master is still queued. ## Risks - Low risk. Test harness only. - A hands-off task that keeps its banner now waits to its deadline before the post-settle click, so a wave with many banners runs longer. - RS1 waves on other workloads no longer get a second end-of-wave click on a task the observer already clicked. That is intended. - Known master CI failures, not touched here: the storage S3 multipart throttle test (being fixed) and the Runner native-distribution-integrity test. - `ci / Runner support suite` failed on head |
||
|
|
7f69ef8b44 |
Prepare and finalize managed workspaces on the active Launcher (#20743)
## Thinking Path > - Paperclip lets plugins own managed Git workspaces. > - A portable Runner task can move to a different physical allocation. > - The checkout must be present on the allocation that launches the Runner. > - Workspace preparation and finalization need exact placement when they retry. > - This change checks the placement at Runner admission and records it before finalization. ## Linked Issues or Issue Description **What existing behavior does this improve?** Workspace preparation and finalization for portable Runner tasks. **Current behavior** A task replacement can change the allocation after Git preparation. A lost finalization reply can leave retry without the original placement. **Proposed behavior** Prepare the checkout on the actual allocation before admission. Record finalization placement so an exact retry can use it. Portable ownership and controller recovery are in [#21066](https://github.com/paperclipai/paperclip/pull/21066). ## What Changed - Reserve the managed workspace before acquiring the portable task lease. - Pass the task placement to the workspace provider. Reuse the described task while placement stays the same and prepare again before admitting a replacement. - Record the initial preparation's run, agent, generation, and optional task placement before dispatch. Inspection and retries on the same Launcher replay the same request; VM and sandbox preparation use the same recovery path. - Keep provider request identity separate from Git credential authority: reservation reconciliation uses the existing project-consent path in a host-only scope bound to the plugin and workspace. Foreground preparation still requires an active run; consent, repository, and revocation checks are unchanged. - Acquire the current Launcher before retrying preparation. If placement changes, use ordinary preparation on the replacement so the provider can settle the prior writer and recover the retained checkout. No additional provider discovery call is added. - Select finalization placement before publishing the operation record, then dispatch and recover using that saved choice. Retain the Launcher during bounded finalization retries. - Preserve interrupted pre-upgrade finalization requests without placement, including their original operation IDs and provider routing. No migration or new durable mode flag is required. ## Verification - Final revision `74ea3e6712`: server typecheck and exact SDK source-pin verification (538 inputs) passed. - All 185 tests passed across the full plugin worker manager, project Git authorization, and plugin workspace integration suites (including 56 workspace tests). Independent verification reran the three lost-reply cases: inspection, same-Launcher retry, and replacement after deleting the original Launcher. The tests assert the correct request identity, durable intent, and clearing that intent after success. - Before the final retry-accounting fix, all 323 tests passed across workspace admission, runtime execution, native workspace finalization, plugin workspace integration, and terminal cleanup retry. Regressions exercise crashes after operation insertion but before dispatch in both finalization paths, exact replay after lease release, and retry after ambiguous placement clears without publishing an incomplete operation. Pre-upgrade interrupted requests retain their operation IDs despite an available portable lease. - Final retry-accounting fix: all 84 native workspace finalizer, failure, and supersession tests passed. A real missing-plugin regression first failed on the prior commit, then verified healthy-run progress, three-attempt exhaustion, terminal cleanup, and absence of a misleading workspace operation. Independent verification also passed the legacy and crash-before-dispatch cases. - Worker-process tests cover both preparation RPC methods: the provider still receives the original run and agent, maintenance receives project-scoped credential authority, and foreground or mismatched-scope requests remain denied. Database tests confirm a stopped run cannot authorize foreground Git access while valid project consent can authorize maintenance. - Existing metadata protection coverage proves callers cannot overwrite the private preparation intent or read it through workspace responses. - Two existing Launcher Git tests passed: retry preserves dirty files and the original base commit without fetching, and retry resumes the saved checkout intent while rejecting a changed pending branch. - Implementation was delegated to GPT-6.1 Sol and independently reviewed. The GitHub stack's automatic rebase preserved the original workspace-placement patch. - [Full CI](https://github.com/paperclipai/paperclip/actions/runs/43293046205/job/131452117706) passed on `74ea3e6712`, including server and workspace tests, Runner, end-to-end tests, typecheck, build, and canary dry run. Ordinary Greptile and [Apex](https://github.com/paperclipai/paperclip/pull/20743#issuecomment-6788920388) both scored `74ea3e6712` 5/5, with no unresolved findings. ## Risks - Live Daytona end-to-end qualification is pending. The checkout record persists on the mounted volume; replacement preparation relies on the provider confirming the previous writer is settled before using that retained checkout. Local tests cover Core request routing and Launcher's durable checkout recovery separately. ## Model Used - OpenAI Codex GPT-6 with GPT-6.1 Sol agents for implementation and independent review. ## Checklist - [x] Thinking path and model are included. - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] The issue is described using the enhancement template. - [x] Local checks passed and new behavior has tests. - [x] All Paperclip CI gates are green. - [x] Greptile is 5/5 with no open findings. --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
ea400ecb07 |
chore(lockfile): refresh pnpm lockfiles (#21178)
## Thinking Path > - Paperclip uses a dependency lock for reproducible builds. > - Dependency changes on master need matching lock entries. > - Paid eval reports also pin a reviewed offline correction. > - Reports must install before paid model work starts. > - This automated PR keeps the lock and report proof together. ## Linked Issues or Issue Description **What happened?** Dependency manifests on master changed after the last committed lock refresh. **Expected behavior** Frozen installs must match those manifests. Eval reports must retain matching offline lock proof. **Steps to reproduce** Run a frozen install before and after this lock refresh. ## What Changed - Refresh the dependency lock from the manifests on master. - Refresh the sandbox provider lockfiles from their manifests on master. - Include the matching offline report correction and hashes. - Resync the SDK public source proof's lock pins from the refreshed lock. ## Verification - Pinned pnpm resolution completed with lifecycle scripts disabled. - The report proof refresh accepted the reviewed committed baseline. - The workflow rejected changes outside the lock and report proof files. ## Risks The lockfile bot merges this PR without CI or review when every added package version is at least 3 days old on npm. Otherwise the PR waits for a maintainer, and the daily refresh tries again. The bot merges only the exact commit this workflow generated from manifests already reviewed on master. Unknown report lock baselines remain blocked. This update changes no permissions or paid enable gates. ## Model Used None — generated by pnpm and the deterministic report-lock updater. ## Checklist - [x] Thinking path and problem description included - [x] Generated lock and matching report proof included - [x] Release-age gate decides between bot merge and maintainer review Co-authored-by: lockfile-bot <lockfile-bot@users.noreply.github.com> |
||
|
|
eaef46b60b |
feat(ui): browser page-load tracing to Honeycomb for hosted instances (#21156)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The board UI is the main surface. Its load speed is a product requirement. > - The team wants to track page-load times over time and give each metric an owner and an SLO in Honeycomb. > - Server traces show only the server part of a page load. They do not show JavaScript start-up, lazy route code, rendering, or client-side navigation. > - Paperclip is also self-hosted by others. Browser tracing must stay off there, and it must not slow any page. > - This pull request adds real-user browser tracing for hosted Paperclip Cloud instances only. It loads lazily and sends a closed set of attributes. > - The benefit is one `page.load` measurement per page (named by route pattern), plus Web Vitals and API spans that join the server trace, ready for Honeycomb queries and SLOs. ## Linked Issues or Issue Description No public issue exists. Feature description: **Subsystem affected** Board UI (`ui/`), the authenticated session route (`server/src/routes/auth.ts`), and the shared session schema. **Problem or motivation** We cannot measure how long the all-tasks page or a task page takes to load for real users. Server traces stop at the API response. Most navigation is client side, and the standard OpenTelemetry document-load instrumentation does not cover it. Without a per-page browser measurement, no agent can own a page-load SLO. **Proposed solution** The server sends a `browserTracing` config in the authenticated session only when the instance is cloud managed and `PAPERCLIP_BROWSER_TRACES_ENDPOINT` is set. The browser then measures every hard load and client navigation in one shared recorder tied to the router's own route tree. After the page settles, it lazily loads an OpenTelemetry chunk that exports `page.load`, `web_vital.*`, and same-origin `/api/` fetch spans to Honeycomb. An allowlist removes every other attribute before export. **Alternatives considered** `@honeycombio/opentelemetry-web` was the first choice. Its defaults add the full page URL, query string, hash, referrer, and user agent to every span and to the resource, and it bundles log and metric pipelines. We would have to replace its exporter and remove most of its defaults. Plain OpenTelemetry web packages plus `web-vitals` are smaller and send only what we choose. Honeycomb needs only OTLP/HTTP and its ingest-key header. **Roadmap alignment** This is internal hosted-instance observability. It changes no product behavior and no self-hosted behavior. ## What Changed - `server/src/browser-tracing-config.ts`: resolves the browser tracing config. It returns `null` unless `isCloudManagedInstance()` is true and `PAPERCLIP_BROWSER_TRACES_ENDPOINT` is set. It accepts only a Honeycomb ingest key (`hcaik_` prefix), requires `https` (except loopback), and fails closed with one value-free warning on any invalid setting. Optional: `PAPERCLIP_BROWSER_TRACES_INGEST_KEY`, `_SAMPLE_RATE` (default 1), `_SERVICE_NAME` (default `paperclip-web`), `_ENVIRONMENT` (default `SENTRY_ENVIRONMENT`). - `GET /api/auth/get-session` adds `browserTracing` (config or `null`), the same delivery path as the front-end Sentry DSN. The shared `authSessionSchema` has the new optional field `browserTracingConfigSchema`. - `ui/src/lib/browser-tracing/page-load-recorder.ts` and `controller.ts` (main bundle, no OpenTelemetry imports): one `page.load` measurement per hard load (from navigation start) and per client navigation (from the history change). A measurement ends at the first painted frame where no full-page loading state is mounted and either every page-declared content signal is ready or no query without data is fetching. Redirects continue the measurement. A new navigation interrupts it. 30 s times out. The sample decision is made once per page session before any download. - `ui/src/components/PageLoadTracker.tsx`: `App` passes the same `<Route>` tree it renders, so every page is reported by route pattern (for example `/:companyPrefix/issues/:issueId`) with no per-page code. - `usePageContentReady` hook and blocking gates: `PaperclipLoading` (also the route `Suspense` fallback) blocks readiness. The task page uses the existing `issue-detail:navigate→content-paint` point. The plain all-tasks list signals when its first rows render. - `ui/src/lib/browser-tracing/sdk.ts` (lazy chunk, loaded after the page settles): OpenTelemetry web tracer, OTLP/HTTP exporter, fetch instrumentation for this origin's `/api/` only with `traceparent` sent only to this origin, `web-vitals` LCP/INP/CLS, and a context manager that makes the open `page.load` span the parent of its API calls. `span-sanitizer.ts` applies the closed attribute allowlist (no URLs, ids, query strings, status messages, or event attributes). Uncaught errors are not sent because Sentry already reports them. - New `ui` dev dependencies, pinned exactly. Each is needed, and the OpenTelemetry ones are the same versions the server already resolves: - `@opentelemetry/api@1.9.1` (already in the workspace): tracer and context API. - `@opentelemetry/sdk-trace-web@2.10.0`: the browser tracer provider, batch processor, and stack context manager. - `@opentelemetry/instrumentation-fetch@0.221.0`: fetch spans and `traceparent` injection. - `@opentelemetry/resources@2.10.0` (already a server dependency): the resource. - `@opentelemetry/otlp-transformer@0.221.0` (already a server dependency): OTLP JSON serialization for a 60-line `OtlpFetchExporter`. This replaces `@opentelemetry/exporter-trace-otlp-http`, which is a server optional peer: installing it in the workspace made the server's exact-version gate test see it as present. - `web-vitals@6.2.3`: LCP, INP, and CLS. All of them are in the lazy chunk only. The lockfile is not committed: `pr-trusted` rejects lockfile edits in pull requests, CI installs with `--no-frozen-lockfile` after the frozen attempt reports `ERR_PNPM_OUTDATED_LOCKFILE`, and the lockfile bot refreshes it after merge. - `tests/runner-e2e/source-sdk-public-inputs.json`: refreshed pins for the three shared files (the session schema is part of the pinned public SDK surface; the contract must live in `packages/shared`). - Docs: new `doc/browser-performance-tracing.md` (enablement, config, key handling, span and attribute list, page-load definitions, example Honeycomb queries and an SLO SLI). Pointers in `doc/observability.md` and the Honeycomb tracing skill. - Two App routing tests mock `PageLoadTracker` because they render `App` without a `QueryClientProvider`. ## Verification - `pnpm verify` returns exit 2 (broader verification required) because docs are global CI inputs. Required CI supplies the broad coverage. - Focused local tests, all pass: - `ui`: `vitest run src/lib/browser-tracing src/components/PageLoadTracker.test.tsx src/components/SentryGate.test.tsx src/api/auth.test.ts src/lib/issue-detail-performance src/pages/IssueDetail src/pages/Issues src/App src/components/CloudAccessGate.test.tsx src/components/IssuesList`: 17 files, 484 tests. - `server`: `browser-tracing-config.test.ts`, `auth-routes.test.ts`, `sentry.test.ts`: 63 passed, 5 skipped. - `packages/shared`: `validators/access.test.ts`: 19 passed. - New tests prove: the SDK module is never imported when the session config is `null`, absent, or signed out, or when the session is not sampled; route templates contain no ids, slugs, or tokens; a client navigation produces a `client` measurement under `/:companyPrefix/issues/:issueId`; `traceparent` goes only to this origin's `/api/` and not to a third-party or non-API URL; exported spans carry only allowlisted attributes. A mutation check (tracker forced on) turned the "never imports" tests red. - Typecheck: `ui` (`pnpm typecheck`), `server` (`pnpm --filter @paperclipai/server typecheck`), and `packages/shared` pass. - Real-browser run: a local dev server with a local OTLP collector and headless Chromium. The self-hosted session returned `browserTracing: null` with the endpoint set, and the page made 0 SDK and 0 collector requests. With a cloud session config injected into the session response, the collector received a hard `page.load` (route `/:companyPrefix/issues`, `ready_signal=idle`), a client `page.load` to `/:companyPrefix/issues/:issueId` (`ready_signal=content`), a back navigation, LCP/INP/CLS spans, and `/api/` fetch spans parented to the open `page.load` span. No task identifier, company id, task title, host, or URL appeared in the exported payload. No third-party request carried `traceparent`. - Bundle size (`ui` production build, initial JavaScript closure of `index.html`): before 3,813,986 B raw / 1,153,991 B gzip; after 3,820,743 B / 1,156,590 B (+6.8 KB raw, +2.6 KB gzip, +0.2%). The lazy SDK chunk is 84 KB raw / 27 KB gzip and is not fetched unless tracing is on and sampled. - Review fixes (each red on the old code, green on the new): the SDK now loads by a 35 s deadline even if the page never settles, so a timed-out hard load is exported (`controller.test.ts`); an agent chat with no saved task uses the idle rule instead of a content gate that never fires (`usePageContentReady.test.tsx`); the sample rate is rounded to exactly 1/N so Honeycomb's `SampleRate` weight matches (`browser-tracing-config.test.ts`). The real-browser run was repeated after the exporter change with the same result. ## Risks - Low risk for self-hosted installs: the server never sends a config, so no tracing code beyond the small recorder runs, and the SDK chunk is never fetched. - On hosted instances the feature is off until an operator sets `PAPERCLIP_BROWSER_TRACES_ENDPOINT` and an ingest key. When on, the controller wraps `history.pushState`/`replaceState` to time navigation starts and restores them on sign-out. - Every signed-in browser can read the ingest key, like the Sentry DSN. Only Honeycomb ingest keys (send-only) are accepted. A collector endpoint can keep the key server-side. - Hard-load API calls happen before the SDK loads, so they have no fetch spans. The hard-load `page.load` span is still recorded. - The browser-to-server trace join needs server tracing. The hosted image does not install the OpenTelemetry server peers today (`CLOUD_BUNDLED_SERVER_DEPS` is `@sentry/node` only). Browser spans work without it. - `ui/src/pages/IssueDetail.tsx` was already over 1,000 lines. This change adds 5 lines there for the task page content signal, next to the existing content-paint measure. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) via Claude Code, with tool use (shell, file edits, local browser automation). Reasoning effort: low. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
edf7edac60 |
test(restart-chaos): keep wake evidence and early run events, and word staging-mix rounds unambiguously (#21173)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness runs agent tasks across server swaps and keeps evidence for each failure > - Wave `w-20261112T181948-s7101` had two "run started after the task was done" cases that could not be traced > - The wave deleted its database before anyone saved which wake started those runs, which server created them, or how the task status changed > - The evidence bundle also kept only the last 200 events of a run, so it dropped the first 291 events of a Retry run, including its wake payload > - The staging-mix wording made one Codex agent refuse to redo an interrupted round, and the task blocked on a harness artefact > - This pull request saves that evidence for every task, keeps a run's first events, and rewords the round rules without changing the grade ## Linked Issues or Issue Description Refs #21125, #21168. **What existing behavior does this improve?** Restart chaos evidence capture and the staging-mix task wording. **Subsystem affected** Test harness only: `tests/restart-chaos/`. No product code changes. **Current behavior** - The harness does not save a task's wake rows, run context snapshots or status history before the wave database is deleted. - `evidence.ts` keeps only the last 200 events per run. - The task tells the agent "do not skip, repeat or shorten any" round, and the follow-up says "do not restart any". An agent read that as "never re-run an interrupted round". **Proposed behavior** - Save, for every task, the wake rows, run snapshots, status changes and serving server generations. - Keep a run's first 400 and last 200 events, with a marker between them. - Say plainly that every round must appear in the ledger exactly once, that an interrupted round must be run again, and that a round already in the ledger must never be run again. **Reason and benefit** Wake and run origins can be traced after the wave, and the wording no longer creates failures that come only from the harness. ## What Changed - New `wake-evidence.ts`, called once at the end of each wave while the database still exists. It writes `wake-evidence/<task>.json` for every task. Each file holds: - for each run: its `context_snapshot`, invocation source, trigger and wakeup request id; - the `agent_wakeup_requests` rows for the issue and for each run: reason, source, trigger detail, requesting actor, idempotency key, payload and times; - the issue's status changes from `activity_log`. Each wake and run names the server generations alive when it was created (`servingAtCreate`): pid, boot id, epoch and process space from `controller_instances`, plus the harness's primary or secondary role during an overlap. Each run also names its owning controller (`owner`). `wake-evidence/servers.json` lists every controller and harness server window. Known secret values are redacted. - `evidence.ts` keeps all of a run's events, or the first 400 and the last 200 with an `omittedEvents` marker between them. - `staging-mix.ts` only changes the wording; the grading is unchanged. The new text: - every round must appear in the ledger file exactly once, with the hash the command printed; - before a round, read the ledger; a round whose line is there is finished and must never run again; - a round interrupted before its line was written must be run again. The follow-up comment says the same. - `main.ts` gains one call; the file stays at 1,000 lines. README updated. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 59 files, 711 tests pass. That includes a real embedded PostgreSQL test that writes a wake, a run, a status change and a controller row, then checks the evidence file. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: passes. - Unit tests cover the serving-generation join during an overlap, the per-issue join, the first 400 and last 200 events of a 1,291-event run, and the new wording. - Not run: a live wave with this change. ## Risks - Low risk. Test harness only. - Evidence capture is one set of reads per wave. A failure is recorded as `wake_evidence_failed` and does not fail the wave. - Known master CI failures, not touched here: the storage S3 multipart throttle test and the Runner native-distribution-integrity test. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, file edits). Extended reasoning was available. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
8ff45c0fa9 |
feat(connections): render MCP Apps views in the task chat, starting with Circleback (#21119)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Agents reach external apps through MCP connections. Paperclip
records each call and shows it as a tool row in the task chat.
> - MCP Apps (SEP-1865, extension `io.modelcontextprotocol/ui`, spec
2026-01-26) lets an MCP server attach an interactive `ui://` view to a
tool. Claude, Claude Code and VS Code render these views in chat.
Circleback uses them to show meeting cards.
> - Paperclip dropped the tool's `_meta.ui` at discovery and the
result's `_meta` at call time. Its gateway reported MCP App UI as
"unsupported". Circleback was reachable only through Composio, whose
toolkit has no `Display*` tools.
> - This pull request makes Paperclip an MCP Apps host. It also adds
Circleback as a native catalog connection.
> - The benefit is that a Circleback recap in a task shows the same
meeting cards as in Claude, and any other MCP server with views works
the same way.
## Linked Issues or Issue Description
**Subsystem affected**
Connections and apps (remote MCP gateway) and the task chat.
**Problem or motivation**
An agent that uses Circleback in Paperclip can only return text. The
same tools in Claude Code show interactive meeting cards
(`DisplayMeetings`). Paperclip discards the MCP Apps metadata, so it
cannot show them.
**Proposed solution**
Paperclip implements the host side of MCP Apps end to end. It keeps the
tool's view declaration and records each call that has a view. It
renders the view under the tool row in a sandboxed iframe that uses the
MCP Apps postMessage protocol. Circleback becomes a native catalog
connection, so its `Display*` tools are available.
**Alternatives considered**
Composio's `circleback_mcp` toolkit has 11 read tools and no `Display*`
tools, and its meta-tools return plain JSON, so it cannot carry views.
The spec's double-iframe sandbox proxy on a separate origin needs a
second origin that Paperclip does not have. A single opaque-origin
`srcdoc` frame with an injected CSP gives the same isolation.
**Roadmap alignment**
This extends the existing Apps catalog and MCP gateway. It adds no new
integration substrate (decision D4 in doc/connections/README.md).
**Additional context**
The standard, the design and the testing plan are in
`doc/connections/MCP-APPS.md`.
## What Changed
- **Shared:** `packages/shared/src/mcp-apps.ts` adds MCP Apps types,
`_meta.ui` and resource parsing (current and legacy keys), and a CSP
builder that keeps only well-formed `https` origins.
- **DB:** migration `0471` (`online_compatible`) adds
`tool_catalog_entries.ui_meta` and a new `tool_mcp_app_views` table.
- **Server, catalog:** discovery keeps each tool's `_meta.ui`. It is not
part of the version hash, so a view never quarantines a reviewed tool.
- **Server, gateway:**
- Paperclip advertises the MCP Apps client capability when it
initializes a session.
- App-only tools (`visibility: ["app"]`) are hidden from agents.
- The result `_meta` is kept for the view and is not sent to the agent.
- Each successful call to a tool with a view records a view row.
Recording is best effort.
- The gateway's `initialize` now reports `paperclip/mcp-app-ui:
host_rendered`.
- The context protocol (`resources/read`) now uses the injected test
transport, like the other remote calls.
- **Server, routes:** `server/src/routes/mcp-apps.ts`
- `GET /api/issues/:issueId/mcp-app-views` lists a task's views. It
needs task read access.
- `GET /api/issues/:issueId/mcp-app-views/:viewId` returns the view
document, read live through the agent's connection.
- `POST /api/issues/:issueId/mcp-app-views/:viewId/tool-calls` runs the
view's `tools/call`. It needs `tools:use` and reaches only tools of the
same connection that allow `app`. It uses the connection test-call path,
so policy, Ask first and audit apply.
- **UI:** `ui/src/components/mcp-app/`
- The host protocol: `ui/initialize` with theme, style variables and
size limits; tool-input; tool-result; size-changed; `tools/call`;
`ui/open-link` (http and https only); teardown.
- An opaque-origin iframe (`allow-scripts`, never `allow-same-origin`)
with the CSP placed before the view's markup.
- The views of a run render with that run's reply, in the live turn and
in settled history. They are keyed by the reply's run, so history needs
no transcript.
- **Circleback:** a native catalog app (`mcp-oauth`, DCR, `user
user:write`) with:
- the official logo;
- a reviewed risk classifier (Share* tools are writes, Delete* tools are
destructive, Display* tools are reads);
- warnings in the sign-in step;
- `doc/connections/CIRCLEBACK.md`.
Search now prefers a native app over an aggregator listing named "X
MCP".
- **Storybook:** `Tasks/MCP App views` shows the tool row with its view,
the ready state (Refresh calls a tool), loading, view unavailable, and a
tool call that needs approval.
## Verification
- `pnpm exec vitest run packages/shared/src/mcp-apps.test.ts
ui/src/components/mcp-app/`: 19 passed.
- `pnpm exec vitest run server/src/__tests__/mcp-apps.test.ts`: 3
passed, on embedded Postgres. The test connects Circleback through the
catalog with DCR OAuth against a fake Circleback MCP server. It checks:
- `ui_meta` is stored;
- an app-only tool is hidden from the agent;
- an agent call records one view with arguments, structured content and
`_meta`;
- a call without a view records nothing;
- the routes serve the view's HTML and CSP;
- the view's tool call works and a model-only tool is refused;
- another task, another company and an agent actor are refused.
- Circleback connector: 13 focused files, 645 tests passed. These are
the app definitions, search, aggregator fallback, connection intents,
tool policy, `AppsConnect` and `Browse`.
- Typecheck passed for `@paperclipai/shared`, `@paperclipai/server` and
`@paperclipai/ui`. `node scripts/storybook-catalog.mjs` and the
migration compatibility check also passed.
- Storybook check in the browser:
- In both themes, the view runs the handshake, renders the meeting cards
from `structuredContent`, picks up the theme variables, and resizes to
its content.
- Refresh sends `tools/call` and re-renders the cards.
- **Live Circleback end to end** on a local instance, with a real
Circleback account:
- Connected through the catalog entry with Circleback's dynamic client
registration and OAuth.
- Discovery stored the real view declarations:
- `DisplayMeetings` uses `ui://circleback/meetings.v2.html`.
- `DisplayActionItems` and `DisplayCalendarEvents` also have views.
- `ListAssigneeOptions` and `SetCalendarEventRecording` are app-only
helpers with no view. They are hidden from agents.
- A `claude_local` agent called `DisplayMeetings` on two tasks.
- The real 3.5 MB Circleback view rendered the meeting cards with the
agent's reply.
- Clicking a card opened that meeting in Circleback through
`ui/open-link`.
- The live run found two bugs, both fixed in this PR, each with a
regression test:
- The 1 MB tool-result limit blocked the view document.
- Views were hidden inside the folded tool history.
## Risks
- **New migration:** a new table and a nullable column. It is
`online_compatible`.
- **Existing connections:** tools get `ui_meta` on the next catalog
refresh. Until then, no view shows and the tool row is unchanged.
- **Untrusted provider HTML:** the view runs as provider HTML in an
opaque-origin sandbox. It cannot reach Paperclip's session, DOM or API.
Its network access is limited to the origins its server declared. A
view's tool calls use the viewer's permissions and the normal policy and
Ask-first path.
- **Agent tool lists:** app-only tools are removed from them. No current
provider marks tools app-only, so no listed tool disappears.
- **Search:** the change that prefers a native app over an "X MCP"
aggregator listing affects every app.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and
subagents (one subagent authored the Circleback catalog connector).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
6a24b417c0 |
test(restart-chaos): add hosted mode for the release-swap proof on a real staging instance (#21153)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Hosted instances take an automatic release about every 40 minutes as an overlap swap: a new server starts beside the old one, both serve for a while, then the old one stops > - The restart-chaos harness (`tests/restart-chaos/`) proves release swaps only on servers it starts locally, with the release-swap cell RS1 (#21126) and the staging-mix workload (#21125) > - The zero-downtime finish line needs the same proof on a real hosted staging instance across real releases, with the same per-task grade > - This pull request adds a hosted mode: the harness starts no server and injects no fault; it runs the staging mix against one allowed hosted instance through its API, detects each real release from `/api/health`, and grades every started task with the RS1 grade module > - The benefit is one comparable number per hosted run (`tasks completed without human action / tasks started`) plus per-release continuity, with no database access and no risk to other instances ## Linked Issues or Issue Description Refs #21126 Refs #21125 Refs #21139 **Is your feature request related to a problem? Please describe.** The release-swap proof runs only against local servers. It cannot show what a real hosted instance does across its own automatic releases. **Describe the solution you'd like** A hosted mode in the harness. It takes one explicitly allowed staging or test host, signs in with an existing board token, creates labelled test agents and tasks, watches real releases through `/api/health`, grades every task with the RS1 grade, reports web and API continuity, and removes what it created. **Describe alternatives you've considered** Reading the hosted database directly. That needs production-grade database credentials and gives a second, different collector. The API path uses the same access a person has and reports the fields the API cannot supply as unavailable. ## What Changed - `hosted-guard.ts`: refuses to start unless `--hosted-base-url` is a bare `https://` origin and `--hosted-allow` names the same host. Only one stack under `.staging.paperclip.app` or `.test.paperclip.app` is accepted. The shared staging instance, production, the fleet admin and Cloud hosts are always refused. Every request is checked against that origin before it is sent. - `hosted-api.ts`: a bearer-token API client (token from `PAPERCLIP_HOSTED_TOKEN` or the CLI store, never printed) that logs every request for continuity, retries only reads, and a `/api/health` probe. - `hosted-release.ts`: pure release detection over health samples. It reports each release's overlap window (two process identities answering), failed and slow requests, time to the first answer from the new build, its shape (`overlap`, `stop_then_start`, `clean_switch`) and probe continuity (requests, failures, longest gap). - `hosted-readers.ts`, `hosted-grade.ts`: API-backed readers for runs, tasks, task views, pending questions and approvals, failure chains, next events and duplicated comments and subtasks. The grade calls `gradeReleaseSwapTask`, `summarizeGrades`, `swapMetrics` and `releaseSwapMarkdown` from `release-swap-grade.ts` (#21126), clicks Retry where the board offers it, and labels each failure. Fields the API cannot supply (queued wakes, the owning server process, runs lost while the old server lived, double executors, late-start and preparation probes) are listed as unavailable. - `hosted-setup.ts`: works in the signed-in user's company (Cloud refuses company creation). Creates one labelled project, one labelled agent per adapter and one labelled secret per provider key through the normal APIs. Cleanup pauses and removes the agents, cancels their live runs, hides their tasks, deletes the secrets and archives the project. It touches only names that carry the run label. - `hosted-workload.ts`: drives the staging-mix tasks from `staging-mix.ts` (#21125) at `--hosted-concurrency` (default 5), posts follow-ups during runs, and clicks an offered Retry once per failed run. - `hosted.ts`, `hosted-config.ts`, `hosted-main.ts`: the entry point, options, `--hosted-observe-only` (read-only release watch), `--hosted-cleanup-only`, and the run. `main.ts` and `supervisor.ts` are unchanged. - Hands-off grading (`hosted-handsoff.ts`, `hosted-workload.ts`): no Retry click during the wave or the settle period. Each task records when a Retry banner first showed and whether it cleared by itself. After the settle period the harness clicks Retry once on tasks that are not done and records whether it worked. `SUMMARY.md` starts with `<pass> / <started> tasks completed hands-off`, `Retry banner shown: <n>` and `Retry worked / clicked: <w> / <c>`. `--hosted-retry-during-wave` keeps the old behavior. - Assigned-work invariant: the grade reads `GET /api/companies/:id/assigned-work-invariant` once (from #21139) and reports the company's violation counts and the sampled violations and stalled slots of this run. A 404 on an older build is reported as not available. - `README.md`: a Hosted mode section with commands, the guard, sign-in, setup, detection, grade, unavailable fields and cleanup. ## Verification - `node server/node_modules/vitest/vitest.mjs run --config tests/restart-chaos/vitest.config.ts tests/restart-chaos/hosted-*.test.ts tests/restart-chaos/release-swap-grade.test.ts`: 52 tests pass (38 hosted tests). They cover the host guard (allowed host, mismatched allow flag, the shared staging instance, production, fleet admin, Cloud and look-alike hosts, a userinfo trick in a route), token lookup, release detection (overlap, stop-then-start, a restart on the same commit, unsettled release, continuity), the API readers, the end-to-end API grade with a Retry click and labels, the workload, and label-only cleanup. - `node server/node_modules/typescript/bin/tsc -p tests/restart-chaos/tsconfig.json`: clean. - Run without a token: the harness stops before any request and names the missing sign-in. Run against the shared staging host: refused by the guard. - `pnpm verify` asks for broader CI coverage (exit 2, README is a global CI input); required CI supplies it. - Known failures on master, outside this diff: `server/src/__tests__/storage-s3-multipart-throttle.test.ts` ("halves new parts in flight once when a parallel burst of parts is throttled", expected 3 to be 4) also fails on master and is being fixed separately. `packages/paperclip-runner` `src/drivers/acpx/native-distribution-integrity.test.ts` ("expires a real ACPX permission on native child death without replaying its mutation") is in the Runner package; this PR changes only `tests/restart-chaos/`. - Live smoke on one hosted staging instance (2 tasks): setup, workload, grade and cleanup ran end to end. It showed two harness defects, now fixed: an activity window that was too short for 90 s tool rounds, and agent removal that needed termination first. - Not run here: a hosted run across real releases (none reached the instance during this work). ## Risks - Low risk to the product: harness-only, no product code changes. - The run writes to one hosted instance. The guard, the per-request origin check and the label filter limit it to that instance and to objects it created. - The grade over the API cannot see queued wakes. A task whose only owner is a queued wake grades as having no owner. The report says so. - #21125 and #21126 are on master; this branch merges master and adds only hosted files and the README section. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits). Default context window; no extended thinking mode. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
a924d2e2c7 |
Persist portable Runner ownership through controller recovery (#21066)
## Thinking Path > - Paperclip delegates remote Runner execution to environment plugins. > - A plugin can move a session to another task allocation. > - The Core controller can lose an ownership reply or restart before the Runner starts. > - The source and target need a durable record of the same request. > - This change binds restart and attachment to that request and the original unlaunched task lease. ## Linked Issues or Issue Description **What existing behavior does this improve?** Recovery of plugin Runner sessions on replacement compute. **Current behavior** Core cannot replay a session ownership request from saved lease state during startup. **Proposed behavior** Record the operation on both leases and check the matched ownership receipt before attaching. Preserve the original pre-admission lease across controller restart. Managed environment configuration is in [#21065](https://github.com/paperclipai/paperclip/pull/21065). Related warm sandbox cleanup is in [#20065](https://github.com/paperclipai/paperclip/pull/20065). This layer preserves the same-sandbox fallback from [#21001](https://github.com/paperclipai/paperclip/pull/21001). Workspace placement follows in [#20743](https://github.com/paperclipai/paperclip/pull/20743). ## What Changed - Add the optional `sessionHandoffVersion` capability and the ownership task operation. - Save matching ownership claims and fence source checkpoint access. Replay a pending request on the original target. - Restore a portable session using the durable claim and the current credentials for the same account. Start a fresh conversation when the provider reports an unavailable checkpoint, or when Core's persisted compatibility decision selects a different session after transfer. The latter requires an unlaunched target; subsequent recovery must match the fresh task's saved identity. Both paths retain the workspace transfer and source fencing. - Reuse the original unlaunched task lease after hard or graceful controller restart. After a move has attached, validate the surviving Runner through the saved claim rather than requiring its logical lease ID to equal the new physical lease. - Keep older retained leases without a saved account identity on the existing resume path. - Apply the existing supported MCP-only compatibility rule to warm recovery and predecessor accounting. Persist the capability discovered at admission and reuse it after restart without repeating discovery. - Keep predecessor-retirement fallback scoped to the same physical sandbox. A refusal after a portable move preserves the session instead of requesting a replacement that the plugin driver cannot enforce. ## Verification - On `1eb527be27`, server typecheck (including Runner build and SDK dependencies) passed. - Independent final verification passed 168 tests across portable source recovery, session resume, controller recovery, and task authority; two existing tests were skipped. The full portable source integration suite passed 64 tests. - The new integration follows confirmed ownership through an incompatible instruction change, persisted fresh admission, target-only prepare/auth/submit without the source checkpoint, and reattachment after controller replacement. It also rejects unpersisted session selection, ambiguous target leases, and incorrect saved identities. - Before this final change, independent verification passed 1,667 tests across executor, session resume, recovery usage, remote warm recovery, runtime selection, and runtime execution; the environment lifecycle suite passed 208 tests. - Existing regressions cover previous-release account metadata, supported and unsupported MCP changes, provider/instruction incompatibility, and admission capability reuse without another discovery call. - On `51d7eec373`, all four S3 multipart throttle tests passed independently. CI exposed an existing timing-sensitive test twice; its 20 ms overlap assumption now uses synchronized retries and an event-loop yield. Production upload code is unchanged. Temporary concurrency caps of one and eight both fail the retained exact-four assertion. - Exact SDK source-pin checks passed for both stack heads. The upper #20743 code patch remains unchanged; its documentation preserves both layers' behavior. ## Risks - The handoff protocol is opt-in. Existing plugin leases without portable ownership metadata keep the previous release's resume path. Live attachment still requires end-to-end qualification. An unverified claim blocks startup. Git workspaces prepared on a portable allocation also need the workspace placement layer. ## Model Used - OpenAI Codex GPT-6 with GPT-6.1 Sol agents for implementation and independent review. ## Checklist - [x] Thinking path and model are included. - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] The issue is described using the enhancement template. - [x] Local checks passed and new behavior has tests. - [x] All Paperclip CI gates are green. - [x] Greptile is 5/5 with no open findings. --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
cdefac664b |
test(restart-chaos): add the release swap cell RS1 with user-view grading (#21126)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart chaos harness (`tests/restart-chaos/`) proves that agent
tasks survive control-plane restarts
> - Staging takes a release about every 40 minutes as an overlap swap: a
new server starts beside the old one on the same database, both run for
8–9 minutes, then the old one gets SIGTERM
> - On 2026-11-12 staging failed or interrupted 577 of 1,000 runs across
these swaps, but the harness restarts one server at a time, grades each
task once and never repeats the swap
> - This pull request adds the release swap cell RS1: repeated overlap
swaps on one database, with the outgoing server starting new runs during
each overlap, and one end-of-wave grade that reads each task the way the
board shows it
> - The benefit is one number per wave, tasks completed without human
action over tasks started, that measures what staging users see
## Linked Issues or Issue Description
**Area**
Restart chaos harness (`tests/restart-chaos/`), test-only.
**Problem**
The harness did not reproduce the staging release shape. It had no
repeated swaps, no run started by the outgoing server during the
overlap, and no grade of every task at the end of a multi-swap wave.
**Proposed solution**
Add the `--release-swap` cell (RS1). It schedules K15 rolling redeploys
on a fixed clock, drives probe tasks through the outgoing server,
records per-swap evidence, and grades every task once after a settle
period.
**Why it matters**
The release cadence will not slow down. The harness must catch the
failures that the swap causes before staging does.
## What Changed
- `release-swap-schedule.ts`: the `--release-swap` options and the
`--release-swap-short` preset (3 swaps, 12 min apart, 510 s overlap)
(count 6, spacing 40 min, overlap 510 s, settle 15 min), their expansion
into a K15 overlap storm (adds `--k15-fresh-home --ephemeral-server` on
Daytona), the swap plan and the `ReleaseSwapRun` clock (swap k at k ×
spacing, never while a swap is unsettled; tasks launch until the last
swap settled).
- `release-swap-driver.ts`: the per-swap driver. It creates probe tasks
through the outgoing server's own port when the incoming server spawns,
when it enters startup recovery, and so that a run starts about 20 s
before the outgoing server's SIGTERM. It samples runs and their owning
controller PID every 2 s and records which server started each run. It
has a per-swap plug-in list; a plug-in can choose the checkout the
incoming server runs (for the later mixed-version variant).
- `release-swap-grade.ts`: the pure grade. One verdict per task (`pass`,
`needed_retry` with whether the click worked, `open_no_owner`,
`blocked_non_person`, `waiting_on_person`, `still_working`), the
per-swap numbers (failed runs, runs lost while the old server was alive,
time from SIGTERM to each surviving run's next event), duplicate runs
and duplicated side effects, the one-number summary, and one failure
label per failing task (`nothing pending`, `blocked for a non-person
reason`, `needed Retry` with the click result).
- `release-swap-finish.ts`: the end-of-wave collection. It waits for the
settle period, reads every task with a run from the API and the wave
database, clicks Retry where the board offers it (board UI when served,
else the same API call as the observer), waits again, and writes
`release-swap.json` and `release-swap-summary.md`.
- Small hooks in shared files: `supervisor.ts` calls `swapHook` at five
phases of the rolling redeploy and lets the hook choose the next server
root (the file stays at 1,000 lines); `config.ts`, `scheduler.ts`,
`types.ts` and `main.ts` wire the option, the plan, the clock, launches
during the overlap and the final grade.
- README section for the cell, its options and its grading.
- Unit tests for the scheduler, the option parsing, the grade
classifier, the per-swap numbers and the overlap evidence.
## Verification
- `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`
(58 files, 704 tests pass, after merging master).
- `node scripts/typecheck-test-projects.mjs tests/restart-chaos` passes.
- Local smokes through the wave queue (`--env local --adapters opencode
--workloads A1,T1 --release-swap --release-swap-count 2
--release-swap-spacing-minutes 5 --release-swap-overlap-seconds 90
--release-swap-settle-minutes 2 --concurrency 2 --tasks 40`):
- `w-20261112T151735-s7101` (first head): 10 / 16 tasks completed
without human action; 26 min wall. It showed that native runs had no
owner in the sampler (fixed here).
- `w-20261112T154840-s7102` (owner fix): 8 / 13 tasks completed without
human action; 24 min wall. The late probe's run was started by the
outgoing server 20.0 s before its SIGTERM.
- Both smokes caught the same product failure: runs the outgoing server
hands off at shutdown (`restartSuspendedRunIds`) are classified
`awaitingEvidence` by the incoming server's startup recovery, even while
the outgoing server is still alive, and are never settled (`I2_settled`
reconciliation violations). They stay `running` with no process, and the
runs queued behind them are never dispatched. This PR does not fix it
(harness only).
- `w-20261112T163456-s7103` (probe agents, drain): 28 / 29 completed;
the late probe started on the outgoing server 20.0 s and 20.2 s before
SIGTERM in both swaps.
- `w-20261112T171115-s7104`, with 40 preparation probes per swap
overloaded the probe agent (late probe dispatched by the new server 30 s
after SIGTERM); the cap is now 6 per swap.
- CI notes on earlier heads: `ci / e2e shard (1/8)` failed once in
`task-loading-frame` (UI, outside this diff; re-run). `ci / General
tests (server (11/12))` failed in
`src/__tests__/storage-s3-multipart-throttle.test.ts` ("halves new
parts…", expected 3 to be 4), outside this diff; it is a known master
failure being fixed separately and is not touched here.
- Not run: the Daytona smoke. No published runner image matches this
checkout's content ID; an image build for master is queued.
## Risks
- Low risk: test-only harness code. No product code changes.
- Waves without `--release-swap` keep their plans and behavior. The only
shared-path change is the K15 overlap redeploy, which now calls an
optional hook that is unset in other waves.
- The probe tasks and the sampling add a few API calls and one database
query every 2 s during each swap.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits). Extended reasoning on.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
49550814fd |
fix(runner-e2e): re-read a process listing that fails once during an owned stop (#21152)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Runner end-to-end harness stops the Paperclip server and its launcher through a process tree owner that reads `ps` > - The owner already waits out a briefly uncertain process group, but one failed `ps` listing ended the stop at once > - The stop then fell back to the direct child and always reported `Owned tree cleanup incomplete`, even when the launcher exited normally > - CI showed this as `Owned tree cleanup incomplete: Error: Could not inspect owned process identities` in `server-stop.test.ts` on 5 unrelated branches > - This pull request re-reads a failed listing a bounded number of times inside the same deadline, and names why `ps` failed > - The benefit is that one transient listing failure no longer fails a server stop, and a persistent failure is diagnosable ## Linked Issues or Issue Description **What happened?** `tests/runner-e2e/server-stop.test.ts > lets launcher cancellation join wrapper shutdown without signaling the server twice` (job "Runner support suite", project `runner-e2e-support`) fails intermittently with `Error: Owned tree cleanup incomplete: Error: Could not inspect owned process identities`, thrown from `tests/runner-e2e/process-tree-owner.ts:265`. It failed 6 times on 5 unrelated branches in 48 hours. **Steps to reproduce** 1. The failing inspection is `readProcessTable()` returning `null` (non-zero `ps` exit, `ps` timeout, or spawn error). The CI logs do not say which, because `ps` stderr was discarded. 2. Locally on macOS, 40 plain runs pass (the Linux `ps` failure does not occur on this host). 3. Inject the fault: make `readProcessTable()` return `null` for 15% of calls. 26 of 30 runs then fail with the exact CI error. **Paperclip version or commit** `master` at the base of this branch (CI on unrelated pull requests, 2026-11-10 to 2026-11-12). **Expected behavior** A failed listing grants no authority to signal anything. The stop reads the table again within its deadline, as it already does for an uncertain group, and it still falls back to the direct child when the listing keeps failing. **Actual behavior** `settleUncertainIdentity` only retried the uncertain-identity error. The first failed listing went straight to `incompleteTreeFallback`, which always throws. ## What Changed - `process-tree-owner.ts`: `settleUncertainIdentity` also re-reads after `Could not inspect owned process identities`, up to 3 attempts per step. It does not start a new read when less than one poll interval (100 ms) is left before the step deadline. No signal is sent between attempts. A persistent failure keeps the old fallback, so the direct child still gets its grace period. - `process-tree.ts`: `readProcessTable` takes an optional `onFailure(reason)` callback and keeps `ps` stderr. The owner adds the reason (exit code, signal, timeout, or stderr tail) to the inspection error. - `process-tree-owner.test.ts`: four regression tests. A listing that fails once during a stop no longer fails it. A listing that always fails still falls back to the direct child after 3 reads. No read starts at the deadline. A failed listing reports its reason. ## Verification - Red/green: both new tests fail on master. The first fails with the exact CI message `Owned tree cleanup incomplete: Error: Could not inspect owned process identities`. Both pass with this change. - Loop of the flaky test, 30 runs each, with a temporary local fault injection (`readProcessTable` returns `null` for 15% of calls, not committed): - Before: 4 pass, 26 fail (24 with `Owned tree cleanup incomplete: Error: Could not inspect owned process identities`). - After: 27 pass, 3 fail. In a 20-run recheck, the only failures (2) came from the test's own preliminary `await owner.observe()` at line 166, outside the stop path. The 15% injected rate is far above the real rate. - Without injection: 40 of 40 pass on master locally. - `npx vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/process-tree-owner.test.ts tests/runner-e2e/server-stop.test.ts`: 41 tests pass. - `tsc --noEmit -p tests/runner-e2e/tsconfig.json`: no errors in the changed files. ## Risks - Low risk. This is test harness code. Each step can now wait up to 2 extra 100 ms polls when `ps` fails. No new read starts within 100 ms of the deadline. As before this change, one `ps` read that is already running can still take up to its 5 s timeout. The exact cause of the Linux `ps` failure is not proven yet. The new error text will name it if it happens again. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
eca1472bae |
test(e2e): stop Cloud auth health stub from reading a disposed response (#21150)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The e2e suite proves that a Cloud tenant returns to its task after Cloud sign-in > - `tests/e2e/cloud-auth.spec.ts` "Cloud recovery preserves the task" failed in CI e2e shards on at least 6 unrelated PR branches in 48 hours > - The error was `apiResponse.json: Response has been disposed`, which blocks unrelated PRs and hides real failures > - The `/api/health` route stub called `route.fetch()` and `response.json()` for each health request. A health request that started just before the test ended kept that handler in flight. Context teardown then disposed the fetched response, and the handler failed. > - This pull request reads the real health payload one time through the test-scoped `request` fixture and fulfills each intercepted health request from that snapshot > - The benefit is that no route handler does network work that can outlive the browser context, so the test cannot fail at teardown ## Linked Issues or Issue Description **What happened?** `Cloud recovery preserves the task (https://my.paperclip.app, task|auth)` and `(https://my-staging.paperclip.app, task)` failed intermittently in CI with `Error: apiResponse.json: Response has been disposed` at the `/api/health` route handler. **Steps to reproduce** 1. Route `/api/health` with a handler that awaits `route.fetch()` and then `response.json()`. 2. Start a `/api/health` fetch in the page just before the test ends. 3. The browser context closes while the handler is in flight. Playwright disposes the fetched response, and the handler throws. **Expected behavior** The test passes when the UI starts a health request near the end of the test. **Root cause** Playwright stores `route.fetch()` response bodies on the browser context request context. Context close clears them (`fetchResponses.clear()`), so `response.json()` (or `route.fetch()` itself, with "Test ended") fails in any handler that is still running. The UI fetches health from several components on mount and after reload, so a request is sometimes in flight at teardown. ## What Changed - `tests/e2e/cloud-auth.spec.ts`: get `/api/health` one time through the test `request` fixture, add the Cloud overrides, and fulfill intercepted health requests from that snapshot. The route handler no longer calls `route.fetch()`. - The snapshot also sets `status: "ok"`. A snapshot taken during startup recovery reports `starting`, which would keep the Cloud gate on the starting screen for the whole test. ## Verification - Before the fix: `PAPERCLIP_E2E_PORT=3293 npx playwright test --config tests/e2e/playwright.config.ts tests/e2e/cloud-auth.spec.ts -g "Cloud recovery" --repeat-each=30` gave 119 passed, 1 failed (`route.fetch: Test ended ... while running route callback` in the `/api/health` handler, the same teardown race). - Deterministic red/green probe (temporary spec, not committed): the page fires five `/api/health` fetches as the last step of the test. With the old handler: 15/15 failed with the exact CI error `apiResponse.json: Response has been disposed`. With the snapshot handler: 15/15 passed. - After the fix: `PAPERCLIP_E2E_PORT=3293 npx playwright test --config tests/e2e/playwright.config.ts tests/e2e/cloud-auth.spec.ts --repeat-each=30` gave 150 passed, 0 failed (120 Cloud recovery runs plus 30 self-hosted runs). After the `status: "ok"` follow-up: `--repeat-each=10` gave 50 passed, 0 failed. ## Risks - Low risk. Test-only change. The health payload is now a snapshot taken at the start of the test, not a live read. The test only depends on the overridden Cloud fields, and the other health fields do not change during one test. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), running in Claude Code as an agent with tool use (shell, file edit, Playwright runs). Standard context window, extended reasoning enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
0bac3510db |
test(restart-chaos): add the staging-mix workload for release-swap waves (#21125)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness (`tests/restart-chaos/`) restarts the control plane during real agent tasks and grades each task > - On staging (2026-11-12), 577 of 1,000 runs failed or were interrupted across overlap deploys every 40 minutes > - The harness missed this: its waves run a few short tasks, while staging runs about 30 agents at once with long tool turns, subtasks, comments and Retries during each deploy > - This pull request adds a staging-like workload (`--workload staging-mix`) that runs under any existing restart cell today, and under the coming release-swap cell > - The benefit is one honest number per wave (tasks completed without human action / tasks started) under a load shaped like staging ## Linked Issues or Issue Description **What existing behavior does this improve?** The restart chaos harness workloads. Every current workload is a few short tasks, so a wave never has 30 runs, a queue, long mid-tool turns, subtasks, or comments and Retries during a deploy overlap. **Subsystem affected** Test harness only: `tests/restart-chaos/`. No product code changes. **Current behavior** A wave plans a fixed list of short tasks with one fault each. The harness holds every board action while a fault is in progress, so no comment or Retry ever lands inside a deploy overlap. **Proposed behavior** `--workload staging-mix` plans a mix of OpenCode, Codex and Claude tasks (default 60/25/15). Tasks keep arriving for the whole wave. Each task runs 15–45 minutes of foreground tool rounds, parents create subtasks, and follow-up comments and Retries wait for an overlap window. Each task has a deliverable that the harness checks from the result. **Reason and benefit** Release-swap waves need a workload shaped like staging. A pass then means the work was done, not only that a status says `done`. ## What Changed - `staging-mix.ts`: the options (`--workload staging-mix`, `--mix-*`), the seeded plan (interleaved adapter ratio, arrivals over the whole wave, one fault slot per `--mix-fault-spacing-minutes`), the SM task (rounds of a hashing shell command, subtasks, `DONE-<nonce>` comment) and its oracle. - The oracle recomputes each round's chained POSIX `cksum` hash. It fails a missing, short or wrong round, a repeated round, comment or subtask (duplicated side effects), a follow-up that did not reach the ledger, and a parent that did not read its subtasks' results. - The follow-up and Retry step waits for an overlap window, for at most `--mix-hold-minutes`. A swap driver registers the window with `setOverlapProbe`. Until then, the window is a fault in progress while the server still answers (K15 overlap). Retries go through the observer's existing click path, so R3 still counts them. - `staging-mix-run.ts`: the arrival gate, the end of arrivals, and the resource guard. The guard refuses `--mix-runs` above the local cap, samples the harness process tree's memory and CPU and the host's available memory every 30 s, and holds launches while memory is low. - The three agents' `maxConcurrentRuns` split the runs by the ratio. The harness keeps runs plus `--mix-queue` tasks in flight, so a queue forms in Paperclip. - Small hooks in `config.ts`, `types.ts`, `main.ts`, `observer.ts`, `seed.ts`, `workloads.ts` and `journeys.ts`. `supervisor.ts` is unchanged. - README section "Staging mix", and `staging-mix.test.ts` (18 unit tests). ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 54 files, 657 tests pass. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: passes. - The unit tests compare the TypeScript `cksum` with the system `cksum`, and run the round command in a real shell. - Live smoke and sizing results: see the comment on this PR. ## Risks - Low risk. The change is in the test harness only. Existing waves do not set `--workload`, so their plans and grades are unchanged. - A 30-run Daytona wave needs Daytona vCPU headroom above the default 40 vCPU budget. The org is shared with the live fleet, so `RESTART_CHAOS_DAYTONA_VCPU_BUDGET` must be raised only to the real headroom. - Fault slots fire relative to each carrier task's real launch, so a slot can drift when launches wait for memory or the in-flight cap. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, file edits). Extended reasoning was available. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
782d6bead4 |
Configure managed plugin Runner lifecycle (#21065)
## Thinking Path > - Paperclip runs agents in managed environments. > - An agent can use an explicit environment or the instance default. > - A lifecycle plugin must receive the driver that provisions the agent. > - A managed plugin environment also needs to select the Runner process lifecycle. > - This change gives lifecycle selection and provisioning the same environment lookup. ## Linked Issues or Issue Description **What existing behavior does this improve?** Managed plugin environment provisioning and lifecycle configuration. **Current behavior** Lifecycle calls lack the selected plugin driver. Managed plugin configuration cannot select warm Runner execution. **Proposed behavior** Resolve the environment once per lifecycle operation. Pass the driver key for the plugin that serves it. Parse and persist the managed Runner lifecycle mode. The portable ownership and workspace placement changes are in [#21066](https://github.com/paperclipai/paperclip/pull/21066) and [#20743](https://github.com/paperclipai/paperclip/pull/20743). ## What Changed - Share agent environment resolution between required plugin selection and the lifecycle RPC. - Pass `environmentDriverKey` to the plugin that serves the selected environment. - Allow managed plugin environments to set `warm` or `per_turn` and persist the selected mode. - Advertise `pluginEnvironmentRunnerLifecycleMode` in the release capability manifest so senders can check support before including the field. ## Verification - Built shared types, the plugin SDK, and Runner. Server typecheck and SDK source-pin check passed for the initial layer. - The capability contract test failed before the manifest fix and passed after it. Independent verification passed all 160 tests across managed configuration, plugin environment persistence, lifecycle routing, and environment reconciliation. - Routing coverage includes the instance default, an explicit second driver from the same plugin, and an explicit sandbox. - [Hosted CI](https://github.com/paperclipai/paperclip/actions/runs/43193311862) passed on `f07b192c21`. The first attempt timed out in an unchanged workspace-recovery fixture and then collided with a background recovery action. The exact test and all 44 tests in its suite passed locally; the isolated shard retry passed without code changes. ## Risks - Existing managed configuration can omit the lifecycle mode. Senders must check the target release for `pluginEnvironmentRunnerLifecycleMode` before sending the new field. Publishing this signal does not update external deployment tools. ## Model Used - OpenAI Codex GPT-6 with GPT-6.1 Sol agents for implementation and independent review. ## Checklist - [x] Thinking path and model are included. - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] The issue is described using the enhancement template. - [x] Local checks passed and new behavior has tests. - [x] All Paperclip CI gates are green. - [x] Greptile is 5/5 with no open findings. --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
b1330c98e8 |
fix(evals): grade full cohort 6 false failures on saved state and keep failed-run evidence (#21089)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat catalog evals check that a chat agent does real product
work, and they grade saved state, not reply wording
> - Full cohort 6 (OpenCode with DeepSeek V4.1 Flash on master
|
||
|
|
0cfab73520 |
test(chaos): W2C changes the agent configuration the way the settings form does (#21011)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart chaos re-wake cell W2C changes the agent's configuration
after a run succeeds and before the task is woken again, so the next run
cannot resume the saved session
> - The harness sent a partial `adapterConfig` patch (`{ idleTimeoutMs
}`). On a volume-free server that patch is refused for the Codex agent
with 422 "Volume-free servers require managed agent instructions", so
the configuration never changed and W2C cells only tested W2
> - The agent settings form does not send a partial patch:
`buildAgentUpdatePatch` (`ui/src/lib/agent-config-patch.ts`) sends the
whole merged `adapterConfig` with `replaceAdapterConfig: true`
> - This pull request makes W2C change the configuration the same way
the settings form does
> - The benefit is that the W2C cell exercises the configuration change
on every provider while the partial-patch refusal is handled separately
## Linked Issues or Issue Description
**What is the problem?**
W2C's configuration change is refused with 422 for a Codex agent on a
volume-free server, so the cell does not test what it is named for.
**What is the expected behavior?**
W2C changes `idleTimeoutMs` the way a person would in the agent settings
form, and the change succeeds.
**Steps to reproduce**
Run a Daytona wave with `--ephemeral-server --adapters codex --workloads
W2C --faults K0`. The timeline records `rewake_config_changed` with
`httpStatus: 422` and the check `config_changed_before_wake` fails.
## What Changed
- `tests/restart-chaos/rewake.ts`: W2C reads the agent and PATCHes `{
adapterConfig: { ...existing, idleTimeoutMs }, replaceAdapterConfig:
true }`.
- `tests/restart-chaos/shape-cells.test.ts`: the W2C step test expects
the full-config patch.
- `tests/restart-chaos/README.md`: W2C describes the patch it sends.
## Verification
- `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts
tests/restart-chaos/shape-cells.test.ts` passes.
- `node scripts/typecheck-test-projects.mjs tests/restart-chaos` passes.
- A volume-free Codex W2C wave is queued to confirm the change returns
200 live.
## Risks
- Low risk. Test harness only. The partial-patch 422 itself is a
separate product question and is not changed here.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and
code execution.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
a858016082 |
fix(evals): fix full cohort 5 chat harness races, keyword graders and fixtures (#21055)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat eval catalog runs real agents through chat journeys and
grades the saved results
> - Full cohort 5 (OpenCode with DeepSeek v4.1 flash, master
|
||
|
|
98c18784c6 |
fix(evals): grade full cohort 4 chat cases on saved state and rendered links (#21034)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat eval catalog runs real agents through chat journeys and
grades the saved results
> - Full cohort 4 (OpenCode with DeepSeek v4.1 flash, master
|
||
|
|
15b0746340 |
fix(connections): tell agents a request card can only be resolved by the person it is addressed to (#21027)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - When an agent needs a person to connect a service, it creates a
connection request card. The card is addressed to one person, and only
that person can complete or decline it. The server enforces this
> - Agents learn how connection requests work from the shared connection
guidance (`CONNECTION_INTENT_AGENT_GUIDANCE`). The `get_task_context`
tool returns it as `connectionGuidance`
> - That guidance did not say who can resolve a card. In chat-eval
cohort 4 (`connection-setup.wrong-person-resolves`), another member said
"I will finish Alex's connection request for them". The CEO agent read
only the task context and replied that "completing it there is the right
move"
> - The product still refused the other member, but the agent encouraged
an action that cannot succeed
> - This pull request adds one line to the shared guidance: a request
card is addressed to one person, only that person can complete or
decline it, and the agent says whose action it waits on
> - The benefit is that agents give the correct next step instead of
sending people to a control they cannot use
## Linked Issues or Issue Description
**What happened?**
A company member told the CEO agent "I will finish Alex's connection
request for them." The request card was addressed to Alex. The agent
replied "Alex's Notion connection request is on COM-1, so completing it
there is the right move. I'll leave the connection approval to you." The
server correctly refused the other member's setup, completion and
decline requests (403).
**Expected behavior**
The agent tells the person that the request waits on Alex, and that only
Alex can complete or decline it.
**Steps to reproduce**
1. Have an agent create a Notion connection request card addressed to
Alex on a task.
2. As another company member, tell the agent in chat that you will
finish Alex's connection request.
3. Read the agent's reply.
**Paperclip version or commit**
master at
|
||
|
|
e879a24293 |
test(evals): check the wrong-resolver reply links Alex's request and task-link labels match their targets (#21028)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat eval catalog runs real chat journeys against real agents
and grades saved product state and rendered links, never reply wording
> - Full chat-eval cohort 4 had two cases whose graders passed replies
with a structural flaw that the grader could see but did not check
> - `connection-setup.wrong-person-resolves` promises that chat explains
whose action is required, but its grader had no reply check at all
> - `navigation-preferences.find-project-view` passed a reply with a
link labelled COM-2 that opened COM-1
> - This pull request adds two structural reply checks, with calibration
tests for valid, reworded-valid and broken replies
> - The benefit is that a reply that does not point to the request, or
that sends a person to the wrong task, fails the eval
## Linked Issues or Issue Description
**What happened?**
- `connection-setup.wrong-person-resolves`: another member said "I will
finish Alex's connection request for them". The product refused that
member (403 on setup options, phase, complete and decline) and kept
Alex's card pending. The grader checked only that. The reply's content
was not checked in any form.
- `navigation-preferences.find-project-view`: the saved reply contained
`[COM-2 Launch open website](/COM/issues/COM-1)`. The grader passed it.
**Expected behavior**
- The wrong-resolver reply points the person at Alex's actual pending
request. The card on that page says "Waiting for Alex", which is whose
action is required. The grader checks the link from saved state, and the
browser step follows the reply's own link to that card.
- Every task link in the find-project-view reply opens the task that its
visible identifier names.
**Steps to reproduce**
1. Run the two cases (OpenCode with
`openrouter/deepseek/deepseek-v4.1-flash-1017`) on master at
|
||
|
|
cf1def0ca6 |
feat(inbox): rebuild compact cards and share task review controls (#20961)
## Thinking Path > - Paperclip helps people manage AI agents and their work. > - The inbox collects decisions, requests, and updates for the people in a company. > - People need a clear personal queue and a consistent way to answer each request. > - Repeated headings, inconsistent card actions, and separate review flows make that work harder. > - This pull request connects the inbox UI to the routing foundation in #20959. > - Compact cards use shared task controls and open supporting work in a side panel. > - People can answer, claim, transfer, archive, and inspect outcomes from one place. ## Linked Issues or Issue Description **What existing behavior does this improve?** The inbox cards, personal views, shared task controls, and decision history. **Subsystem affected** UI. This PR targets master and uses the routing foundation merged in #20959. **Current behavior** The inbox repeats headings and counts. Cards have different layouts and action placement. The requester can repeat the signed-in person's name. Review and question controls differ between the inbox and task thread. **Proposed behavior** Open Assigned to you by default. Show one count line above the cards. Use a compact card layout with consistent actions and supporting details. Show the requester's avatar and name only when they differ from the signed-in person. Share question controls, including a radio option for Other that reveals a text field when selected. **Reason and benefit** People can scan their queue and complete requests without losing the surrounding task or creating separate drafts for the same review. **Breaking changes** The default inbox view becomes personal. The all-assignee and shared views remain available. Card close menus distinguish personal archive and snooze from shared withdrawal. **Additional context** Refs #20959, #19363, and #19520. Refs #20268 for related actionable-request work. This stack adds personal routing and shared controls. ROADMAP.md includes Multiple Human Users and Agent Reviews and Approvals. ## What Changed - Add personal, unassigned, company, all-assignee, and archived inbox views. Add claim, transfer, personal close actions, receipts, and history controls. - Use compact decision cards with a single count line, one-pixel borders, and equal space above, below, and to the right of each primary action. - Open artifacts and synchronized confirmation/approval controls in a shared side panel. Share their revision drafts and action state with the task thread. Other requests keep one editable form on the inbox card while reviewing an artifact. - Make Other match the question options and reveal its field only when selected. Apply this to inbox and task question cards. - Hide self-requester labels, space agent avatars, and use the standard canceled status icon for withdrawn requests. - Use connector logos in connector cards. Keep the connector promotion background and sidebar border continuous. - Add company and project decision-owner controls and document the updated inbox behavior. ## Verification - The complete [GitHub Actions CI matrix](https://github.com/paperclipai/paperclip/actions/runs/43141559641) passes on `8d81a75e49b518fff6b6ff4c23dd67979b0b97ee` (Linux/Node 24: typecheck, build, general and serialized tests, Runner support, browser shards, and aggregate gates). Greptile is 5/5 on that exact head with zero open findings, and the security scan passes. - Local UI typechecking passes. Eight focused suites pass 382 tests: 236 inbox/review/question/connector tests plus all 146 composer tests. Regressions passed red to green for attachment links, prompt rendering, single-form artifact review, connector scope copy, accessible headings, and the composer test harness's missing theme provider. The final commit changes only that test harness; production code is unchanged from `59d015833f`. - Browser acceptance on the production implementation confirms preserved selected answers with one editable question form, accessible Markdown prompts, inert image references, accurate connector scope copy, native PDF downloads, image sidebar previews, and Open file requesting the attachment endpoint while preserving the inbox URL. Request reads and file responses were intercepted; no question answers were written. The preview is healthy on the final head. - Visual and interaction checks cover Other radio behavior in both themes and on mobile, the loaded Google Calendar logo, and 20px above, below, and to the right of the primary action across all 23 visible cards. The affected desktop/mobile navigation and sidebar specs pass locally and in the final CI browser matrix. - Real-editor browser checks confirm Ctrl/Cmd+Enter cannot bypass another person's confirmation. Cancel preserves the draft and unlocks silently; Continue submits once. Approval write failures remain visible. - `pnpm verify` returns exit 2 because DESIGN.md requires broader coverage; the final full CI matrix supplies every requested gate. Maps and Storybook catalog structural checks pass. The routing foundation #20959 merged after passing 249 database tests, server typechecking, full CI, Greptile 5/5, and security scanning. Its backend and SDK registry are preserved. - Demo data is retained for testing; database suites and browser specs use separate disposable databases. ## Risks - The inbox uses the routing contracts merged in #20959. Personal filters and history affect shared task entry points; the affected controls are included in local regression coverage and full CI. - Personal defaults and close-menu wording change inbox behavior. Existing resolver permissions still control shared actions. - Shared revision state and live invalidation affect two surfaces. Focused tests cover draft synchronization, action locks, and incoming updates. - Visual changes use the existing design tokens and component primitives. ## Model Used OpenAI GPT-6 through Codex. Used repository tools, code execution, and browser checks. This session does not expose a more specific runtime model ID or its context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal Paperclip ticket id - [x] I have run focused tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Scott Tong <scott@scottsmbpm5max.lan> Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
8cefbe9977 |
test(restart-chaos): add CR6, a plugin stalled during activation at boot (#20983)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart chaos harness (`tests/restart-chaos/`) injects faults
into a real server with Daytona sandboxes and grades each task.
> - The first version of CR4 (#20966) stopped the replacement's Daytona
plugin worker while the server was still activating it. That found a
product gap: an `initialize` timeout marked the plugin `error` for the
life of the process.
> - #20966 moved CR4 to after activation. That fault is still a
separate, real failure, so it needs its own cell.
> - This pull request adds CR6: a K1 restart, then the replacement's
plugin worker is stopped as soon as it forks, during activation at boot.
> - The benefit is that #20982's activation-timeout retry is checked by
a wave, before and after the fix.
## Linked Issues or Issue Description
Refs #20966, #20982
**What happened?**
No chaos cell stalls a plugin worker during its activation at boot. In
wave `w-20261111T232850-s9521`, CR4 hit that case by accident. The
plugin stayed `error`, and every Daytona run on that server failed,
Retry included.
**Expected behavior**
The harness can stall the plugin during activation after a restart, and
it grades the task like the other class R cells.
**Steps to reproduce**
1. Run a wave with `--faults CR6`. Before this change, the harness
rejects it.
**Paperclip version or commit**
master at
|
||
|
|
f1aef0cb58 |
fix(evals): grade full cohort 3 false failures on saved state, read receipts and rendered links (#21008)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat eval catalog runs real chat journeys against real agents and grades saved product state > - Full cohort 3 failed six cases where the product behaved correctly and the eval was wrong > - A wrong failure hides real product failures in the noise, so each check must accept valid behavior > - Each check must still fail the broken product behavior it exists to catch > - This pull request replaces those checks with checks of saved state, read receipts and rendered links > - The benefit is fewer false failures, with no case made easier ## Linked Issues or Issue Description Refs #21000 **What happened?** - `projects.project-work`, `tasks.private-task`, `status-cards.other-authors-card` and part of `task-coordination.delegate-with-context`: the CEO created a task with no assignee, so the CEO got a queued run on it. The CEO then reassigned the task. The server cancelled the queued run before it started (`issue_reassigned` or `issue_assignee_changed`). The early poll reported this as "Observed provider run failed", and the run policy counted a never-started legacy run as a failed native run. - `budgets-costs.repair-cost-recording`: after the retry, the chat run re-read the original run record with `saveResponse`. The check accepted only an exact whole-input digest, and this input had another shape. - `secrets.personal-secret`: the reply linked My secrets and said "Enter the value only there, never in chat". `solicitsChatCredential` read "never in chat" as a request to send the value in chat. - `companies.import-preview`: the reply linked `/COM/agents/morgan-package`. The check accepted only `/agents/<uuid>`. - `cases.close-case`: the closure was correct. The result turn said "Case COM-C1 is saved as closed". A keyword pattern did not match that sentence. **Expected behavior** Each check passes the recorded valid behavior and a reworded valid variant. Each check fails the broken behavior. **Steps to reproduce** Grade the retained cohort-3 evidence with the old checks. They fail. The calibration tests in this pull request use the same recorded shapes. ## What Changed - Run policy: a new rule admits a queued run that the server retired before it started because its task changed assignee. The run must have no start, no log, and positive server evidence (`reassignmentStopConfirmed` with `providerWorkStarted: false`, or the stale-queued gate). The task must not be the source chat. A later run by another agent on the same task must exist. The run stays in the raw ledger and in the evidence (`supersededQueuedRunIds`). It does not count toward the provider-run bound, because it did no provider work. A case can have at most one such run (`catalog-superseded-queued-runs`). The early poll defers it to the grade instead of failing at once. Cases read the ledger without it. Other cancellations stay failures. - `budgets-costs.repair-cost-recording` (v5): a re-read of the original run also counts when the product saved its response. The `call_api` `saveResponse` read creates an asset that the product logs as `asset.created` by that run. The harness reads the saved asset and requires that it is the original run record in this company. - `solicitsChatCredential`: a chat destination counts only when it is not itself negated ("never in chat", "not here", "rather than in chat"). This is a scope rule, not a new phrase exception. "Paste it here" and mixed requests still fail. `secrets.personal-secret` is v18. - `companies.import-preview` (v3): new `agentRouteNames` accepts `/<prefix>/agents/<id or url key>[/<tab>]` that names the imported agent. It rejects the existing Morgan, other companies and other origins. - `cases.close-case` (v2): the result turn is graded on its own saved replies. Their run must read the saved case (`read_case` or `GET /api/cases/{id}`, bound by the redaction-safe receipt). The reply must render a link to that case. The case must be done. The wording is not matched. `api-read-receipts` now binds the dedicated `read_case` tool. - The four reassignment cases get new oracle versions. Descriptors are updated. No scenario yaml changed, so the TestOps manifest does not change. ## Verification - `npx vitest run --config tests/runner-e2e/vitest.config.ts` on the 17 affected test files: 516 tests pass. The files include `chat-catalog-run-policy`, `chat-flow`, `non-execution-wake`, `chat-catalog-consistency`, `chat-catalog-fingerprint`, `identity-accounting`, `external-credential-language`, `external-personal-secret`, `identity-portability`, `integrator-case-close`, `api-read-receipts` and the other secret-language users. - Each change has calibration tests: the recorded valid shape, a reworded valid shape, and broken shapes (no successor run, a started or other cancellation, churn; another run's saved record; "paste it here"; the existing Morgan's route; an unread or unlinked result turn). - `npx tsc -p tests/runner-e2e/tsconfig.json --noEmit` passes. - `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check` reports no change. - One paid run on a combined branch with the companion harness pull request. The result is in a comment on this pull request. ## Risks - The superseded-run rule trusts the successor run as proof of reassignment. A case oracle still grades the task assignee and the successor's own outcome. - The close-case result turn no longer checks the reply's words. A reply that reads the closed case and links it but states the wrong result passes this one check. The first-turn premature-claim check and the negative-case check do not change. - The credential grader change also affects other secret cases. It only removes the negated-destination false positive. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
b033cc124d |
fix(evals): repair private human steps for webhook, device and key entry; name failed private steps (#21012)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat evaluation catalog checks agent replies and then the human step that follows, such as a person who reveals a key or approves a device > - Some human steps run in a separate private browser process, because they handle credentials. CI keeps the output of that process private on purpose > - Four cases failed only in that private step, so CI showed one fixed message and no cause > - I ran each private step locally against a dev server, where its output is visible, and found the cause of three failures > - This pull request fixes the harness and two product defects, and adds a closed failure record so that the next CI failure names its step > - The benefit is that these human steps work, and a private-step failure is diagnosable without any credential in logs ## Linked Issues or Issue Description Bug report (following `.github/ISSUE_TEMPLATE/bug_report.yml`). **What happened?** - `routines.webhook-trigger`: the private person opens the routine overview. The trigger cards are only in the Triggers section, so the step waits for a missing button. An unhandled response wait then stops the process. - `connection-setup.secure-api-key`: a key-protected MCP server answers 401 without a challenge header. Paperclip shows "Remote app returned HTTP 401" and does not ask for a key. The person cannot enter the key in the normal flow. - `assistant-connections.device-login`: an expired device code page shows "This item doesn't exist or was moved." The server sends "This code is expired, already decided, or invalid." in the OAuth `error_description` field, and the UI drops it. - `onboarding.new-company`: the private step reads back all saved state and then fails one proof check. CI does not say which check. **Expected behavior** - The person reveals the webhook key in the Triggers section and sends authenticated events. - A pasted MCP link that answers 401 shows the key field. - An expired device code says that the code is expired. - A private-step failure names its fixed step and error class, with no value. **Steps to reproduce** 1. Start a local dev server. 2. Create a routine with a bearer webhook, and run the private webhook worker. It times out on the "Edit webhook" button. 3. Paste the address of an MCP server that answers a plain 401 into Connectors > Add by link. No key field appears. 4. Open `/mcp-device?user_code=<expired code>`. The page shows generic not-found text. **Paperclip version or commit** `master` (reproduced on origin/master before this change) **Deployment mode** Local dev server (embedded PGlite) and the CI chat-eval runner **Installation method** Source checkout (`pnpm dev`) ## What Changed - `integrator-webhook-private-worker.ts`: open `<routine>/triggers` (also for the setup-resume link), and attach a handler to the early response wait so a failed click reports its real step. - `server/.../catalog/discovery.ts`: a 401 with no `WWW-Authenticate` header, to a connection that sends no credential, is handled like a challenge without discoverable sign-in (`oauth_challenge`). The existing UI guidance then asks for a key. A 401 to a connection that already sends a credential keeps the old error. - `ui/src/api/errors.ts`: read the OAuth `error_description` field as a readable server message. - New `private-child-failure.ts`: a private child records only a fixed step name and an error class from a fixed list. The parent echoes a step only when it is in the caller's closed list. Error text, stdout and stderr stay private. - The webhook, private browser and identity children use it. Each has a closed step list file. - `identity-new-company-native-proof.ts`: each proof check has a fixed reason name. The private child reports `new-company-proof-<reason>` when a check fails. The accepted conditions do not change. - Tests: server red/green test for the bare 401, UI red/green test for `error_description`, unit tests for the failure record (no text leaks, closed list only, extra fields dropped). ## Verification - Local dev server (embedded PostgreSQL), private workers run directly with the real child process: - Webhook, ready and setup-pending triggers: before, timeout on "Edit webhook". After, receipt with invalid status 401, valid status 202 and a linked task. - Secure key entry with the synthetic notes provider: before, timeout on the key field. After, an active connection with health `ok` and 2 authorized `tools/list` calls. - Expired device page: before, `at step device-expired-page (TimeoutError)`. After, `expiredPageVisible: true`. - `server`: `vitest run src/__tests__/tool-access-service.test.ts -t "pasted link answers"` (fails before the discovery change with 502, passes after). - `ui`: `vitest run src/api/errors.test.ts` (fails before, passes after). - `pnpm test:e2e:runner:typecheck`, `pnpm test:e2e:runner:unit`, server `tsc --noEmit`. - `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD sha> --check`: unchanged. - One paid run of the four cases on this branch. Results are in a PR comment. ## Risks - A plain 401 from an MCP link now returns 422 `oauth_challenge` instead of 502. The draft connection is removed, as for other challenges. Agents that call the connect API see the "needs sign-in" answer instead of a raw status. This only applies when the connection sends no credential. - Some API errors that carry `error_description` now show that text instead of generic copy. That text is written for people. - The failure record is opt-in per child and contains only constants. The parent ignores a record with any other field. - `onboarding.new-company` is not fixed here. Its native hello probe cannot start locally on macOS, so the failing check is not yet known. This change makes the next run name it. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell, file edits, local browser automation through Playwright). Extended reasoning was on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
5a7378af56 |
fix(heartbeat): re-drive a queued run whose dispatch signal was lost after a restart (#20986)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The heartbeat queue starts queued runs. The run-dispatch worker re-runs the queue only when a write sends a "queued run dispatch" signal. > - A follow-up run can queue while its predecessor still holds its sandbox lease. The claim then declines the run without a log line, and the run stays `queued`. > - After a controller restart, nothing sent a new signal once that lease was settled. The run stayed `queued` forever. The issue had no Retry, and the reconciler counted the queued run as a valid next action. > - This pull request sends a dispatch signal from the native finalization writes that can clear a predecessor's cleanup gate, logs why a claim was declined, and bounds the wait. The dispatch worker re-drives an undispatched queued run after 60 s as a backstop, and the ownership reconciler reports and re-drives one still queued after 5 minutes. > - The benefit is that a follow-up starts within seconds of its predecessor settling, a lost signal delays it by about a minute instead of forever, and a run that still cannot start is visible. ## Linked Issues or Issue Description Refs #20909, #20935, #20953, #20970 **What happened?** Restart-chaos wave `w-20261111T223714-s9512` (master `6154b9485e` + #20907 + #20909, Daytona warm, Codex, W2, CR1 with a 360 s hold, `--ephemeral-server`, seed 9512), task t02. Run `0b95b1cd` crossed the outage with an accepted result. The next controller finalized it at 22:47:59Z. Its continuation run `1ad79418` was queued at 22:48:00Z (`retryOfRunId` = `0b95b1cd`, wake `d967700c`, source `native_status_decision`). The run stayed `queued` with `claimed_at = null` until the wave ended 19 minutes later. The agent stayed `running` with no running run. The task had no Retry. The grade was `R1:no_rest_within_budget` and U1 `no_owner_no_action ... queued 485s`. **Root cause** `claimQueuedRun` (`server/src/services/heartbeat/queue.ts`, `lockIssueExecutionClaim`) declines a queued run while `hasPendingRunCleanup` (`server/src/services/execution-blocker.ts`) is true for its predecessor. After the restart, the predecessor's warm `reuse_by_environment` lease was still `active` when the continuation queued, so the first claim was declined. The decline is correct, but nothing owned the retry. `registerRunDispatchWork` (`server/src/services/heartbeat/dispatch-work.ts`) has `next: async () => null`: the queue is only re-run by a later dispatch signal. At wave end the lease was `retained` with `cleanupStatus: success`, which does not block a claim. A dispatch pass in that state starts the run (verified locally). So no pass ran after the gate cleared. The ownership reconciler listed the run as `queued_run` with owner `run_dispatch` (always "alive"), and I7 treats any queued run as a next action, so nothing reported it. This is not the #20935 throttle. The wave base predates #20935, and a throttled intent keeps its wake intent `queued` with a `nextAllowedAt` deadline. Here the run row existed and its claim was never retried. On current master #20935 parks this specific `response_wake` self-wake, but any continuation, comment wake or Retry that queues behind a settling predecessor after a restart has the same gap. The red test uses a `same_agent` continuation, which master still enqueues. **Steps to reproduce** `server/src/__tests__/heartbeat-native-restart-suspension.test.ts`, "starts a continuation that queued behind its predecessor's sandbox release after a graceful restart". ## What Changed - Event-driven dispatch. A lease release already signals queued-run dispatch (`environmentsSvc.releaseLease`, `writeEnvironmentCleanup`, stop-and-retain settlement). I enumerated every `environment_leases` status writer: each one that moves a lease out of a blocking state already signals. The writers that do not signal only acquire a lease or set `pendingCleanupInFlight`, so they add a block and do not clear one. The native coordinator side had no signal, so: - `notifyNativeFinalizationWork` (the shared notifier of coordinator commits and terminal transitions) now also signals queued-run dispatch for the company. - The coordinator writes that clear `leaseOwner` or end the phase outside that notifier now signal in the same transaction: `releaseLapsedCoordinatorOwner`, native startup cleanup, observed-workspace preservation, `record_finalization_error` in the interaction bridge, the wake-queue terminal recovery, and the reconciler's ambiguous-state terminal failure. - Not changed: three non-transactional pre-dispatch closes in `run-completion.ts` and `agent-execution-stop.ts`. Their callers queue or settle the successor after the write, so the successor's own enqueue starts the queue. - `claimQueuedRun` now logs the reason when it declines a queued run at the issue execution claim (`predecessor_cleanup_pending`, `predecessor_still_executing`, `issue_execution_owned_by_another_run`, case handoff, task chat cleanup), when a predecessor is still settling in this process, and when a temporary budget hold keeps it queued. - New `server/src/services/queued-run-dispatch-liveness.ts`: `listUndispatchedQueuedRuns` reads queued runs in active companies whose ready, non-paused agent has no running run. It reads every page with a keyset cursor until the set is exhausted, so older held runs cannot hide a newer stranded run. `createQueuedRunRedriveSchedule` re-drives each such run 60 s after it was created, then every 2, 4, 8 minutes, capped at 15 minutes. Run ids that leave the set are dropped. - `server/src/services/heartbeat/dispatch-work.ts`: the queued-run dispatch worker's `next` now plans those re-drives. It re-reads only the companies the pass just dispatched, keeps the deadlines of every other company, and runs no query when the pass dispatched nothing. Only an instance-wide pass (startup or explicit wake) reads every company. It adds each due company to the dispatch scopes and returns the next due time. - `packages/db`: new partial index `heartbeat_runs_queued_created_idx (company_id, created_at, id) WHERE status = 'queued'` (migration `0470_heartbeat_runs_queued_created_index.sql`, `CREATE INDEX CONCURRENTLY IF NOT EXISTS`, registered as a top-level migration in `client.ts`). It returns `null` (no deadline, no SQL until a signal) when no run is undispatched or scheduling is suppressed. Every re-drive goes through the normal `startNextQueuedRunForAgent` and `claimQueuedRun` gates, so it never bypasses a hold, budget, pause or cleanup gate. - `server/src/services/heartbeat.ts`: wires `listUndispatchedQueuedRuns` into the worker (two lines). - `server/src/services/recovery/ownership-invariant-report.ts`: new check. A queued run still undispatched 5 minutes after creation is an I7 violation (`kind: queued_run`, `reason: queued_run_not_dispatched`), written to the activity log once. A younger one sets the report's next due time. - `server/src/services/native-runtime/native-finalization-reconciler.ts`: `applyOwnershipReportBackstops` sends a queued-run dispatch signal once, for each company with a newly found violation. Repeat passes do not signal again, so a held run does not wake the report every 30 s. - `tests/restart-chaos/README.md`: records the wave and the fix in the CR results table. The existing CR1 cell (#20909) and the enforced U1 check already cover this class in the harness. ## Verification - RED on master (`23909077ad`): - `heartbeat-native-restart-suspension.test.ts` "starts a continuation that queued behind its predecessor's sandbox release after a graceful restart": the continuation is `queued` behind the predecessor's `active` warm lease. The run-dispatch worker is registered, the lease then settles to `retained`/`success` with no dispatch signal, and the run stays queued: `AssertionError: expected null not to be null` after 75 s. - `ownership-reconciliation-report.test.ts` "reports and re-drives a queued run that no dispatch started while its agent had nothing running": `expected [] to deeply equal [ ObjectContaining{…} ]` (no violation for a run queued 19 minutes). - `ownership-reconciliation-report.test.ts` "signals queued-run dispatch when finalization work settles or a finished coordinator's lapsed owner is released": `expected [] to include 'settled'`. - GREEN on this branch: all pass. The restart test has two cases. When the lease is released through `releaseLease`, the continuation starts within 5 s. When the lease is settled by a write with no signal, the backstop starts it within 75 s. A new report test puts three older held runs ahead of a stranded run with a page size of 2, finds the stranded run, and checks that the backstop signals once. - `queued-run-dispatch-liveness.test.ts`: schedule bound, doubling, cap, and going quiet. - Focused suites pass (22 files, 495 tests): recovery-deadline-acceptance, live-recovery-work, heartbeat-run-dispatch-lifecycle, heartbeat-run-dispatch-work, deferred-execution-receipt-work, heartbeat-native-cleanup-admission, the ownership kinds suites, and others. `heartbeat-native-restart-suspension` and `ownership-reconciliation-report` pass in full. `tsc --noEmit` for the server passes. ## Risks - A queued run that is held on purpose (for example a temporary budget hold or a case handoff waiting for release) is now re-checked at most every 15 minutes after the first minute. Each re-check is one indexed query and the normal claim gates. Such a run is also reported once as an I7 violation. - An instance with an undispatched queued run keeps a dispatch deadline, so it does not idle while that run waits. An instance with no such run is unchanged. - Schema: one additive partial index, built concurrently. No data change. State saved by an older version is read as is. Nothing runs before `turn.start`. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
bf18b348c5 |
fix(evals): wait for saved, rendered replies and settled forms, and step through multi-question cards (#21009)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat eval catalog drives real chat journeys in a browser and grades saved product state > - In full cohort 3, four cases acted on the page before the page was ready, or did not step through a card > - The product was correct in each case, so these failures hide real product failures > - This pull request makes the harness wait for the saved, rendered reply and for a settled form, and step through multi-question cards > - The benefit is fewer false failures, with the same product requirements ## Linked Issues or Issue Description Refs #21000 Refs #21008 **What happened?** - `companies.fast-responses`, `agents.choose-primary`, `budgets-costs.set-budget`: the server recorded the chat run as finished 0.2 to 1.3 seconds before it saved the reply comment. The harness read the reply links right after the run finished. It found no reply, so it found no links. - `budgets-costs.set-budget`: the budget card remounted after the harness typed the new amount. The remount reset the draft, so the Update budget button stayed disabled until the click timed out. - `questions-and-approvals.necessary-question`: the agent asked the required product-name question and one optional question. The card showed Next on its first page. The harness only clicked Submit answers. The grade also required exactly one question on the card. **Expected behavior** The harness reads a turn's links after its reply is saved and rendered. It saves the budget only on a settled form. It pages through a card, answers its one required question and skips optional ones. **Steps to reproduce** Run the three link cases when the reply is saved after the run finishes. Run set-budget when the budget card reloads after typing. Run necessary-question when the card has an optional second question. ## What Changed - `chat-rendered-links.ts`: new shared `waitForSavedTurnReplyRender`. It waits (bounded, 15 s) until each finished lead run's reply is saved, then uses the existing render wait for the latest reply. It returns `unsaved` when no reply arrives. The caller then fails with its own error. - `identity-support.ts`: new `waitForTurnReplyLinks` uses that wait for the source conversation. `companies.fast-responses` (v3), `agents.choose-primary` (v4) and `budgets-costs.set-budget` (v3) call it before they read reply links. - `budgets-costs.set-budget`: `settledInput` resolves when the same input node stays on the page for one second. The harness types and saves on that node. If the card still remounts and drops the amount, it types it again, at most three times. - `questions-and-approvals.necessary-question` (v2): the harness pages through a multi-question card. It answers the one required question and clicks Skip on each optional question. The grade now requires exactly one required question, and that question must ask for the product name as free text. Other questions must be optional. An optional necessary question, a second required question, or a required question about something else still fails. - Descriptors are updated. No scenario yaml changed, so the TestOps manifest does not change. ## Verification - `npx vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/chat-rendered-links.test.ts tests/runner-e2e/chat-catalog-consistency.test.ts tests/runner-e2e/chat-catalog-fingerprint.test.ts tests/runner-e2e/chat-catalog-domains/identity.test.ts`: 100 tests pass. - New tests: a reply saved after the run finished is waited for and then rendered; a run that saves no reply returns `unsaved` within its bound. The necessary-question grade accepts the recorded card (required name plus optional follow-up) and rejects the broken cards above. - `npx tsc -p tests/runner-e2e/tsconfig.json --noEmit` passes. - `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check` reports no change. - One paid run on a combined branch with #21008. The result is in a comment on this pull request. ## Risks - A turn whose run saves no reply now waits up to 15 seconds before the case continues. Only the three named cases call the wait. - `settledInput` marks the input node with a `data-e2e-settle-mark` attribute to detect a remount. It does not change product behavior. - #21000 changes the render-wait expiry in the same file. Whichever merges second rebases. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
601babe322 |
fix(chat): let agents read saved trace export status and give chat real company page paths (#20998)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - People talk to their agents in chat, and agents answer from the
company's saved settings
> - A board member can save a company trace export (an OTLP collector or
Braintrust) in Company settings
> - The settings route is for board connection managers only, so an
agent cannot read it
> - In a chat eval, the board saved and tested a collector. The agent
checked only its own config and secrets, and then told the person that
nothing was configured
> - Chat guidance also names company pages as `/<prefix>/...`. In
another eval, the agent copied that placeholder into a link, and the
link opened nothing
> - This pull request gives agents a read-only status of the trace
export, and gives chat the real company prefix in page paths
> - The benefit is that agents answer from saved settings, and the links
they write open real pages
## Linked Issues or Issue Description
**What happened?**
1. A board member saved and tested a trace export destination. The chat
agent was asked to acknowledge the saved configuration. The agent said
that no collector endpoint or credentials existed. Agents get 403 from
`GET /api/companies/{companyId}/observability`, and no other route
reports the saved state.
2. An agent reply linked `/<prefix>/company/settings/instance`. The chat
rendered it as `/%3Cprefix%3E/...`, which is not a real page. The
placeholder came from the chat directive, which names pages such as
`/<prefix>/company/settings/instance/profile`.
**Expected behavior**
1. An agent can read whether the company's trace export is configured,
enabled and credentialed, and the result of its latest test. The agent
cannot read the destination or any credential.
2. The chat directive and API search results that an agent reads give
real company paths, such as `/COM/company/settings/instance/profile`.
**Steps to reproduce**
1. As a board member, save and test an agent observability destination
in Company settings.
2. In chat, ask the CEO agent to acknowledge that the test collector
configuration is saved.
3. See that the agent cannot read the saved settings and denies them.
**Paperclip version or commit**
master at
|
||
|
|
65ff80a4bb |
fix(evals): grade chat acknowledgements, saved replies and links on saved state (#21000)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat eval catalog runs real chat journeys against real agents and grades saved product state > - A review of full cohort 3 found three cases whose oracle passed wrong behavior, and eight passes whose final evidence saved no rendered links > - `companies.agent-observability` passed although the agent denied the saved collector. `inbox-search.show-my-work` counted links that existed only in run-stream prose. `instance-operations.restart-service` accepted a reply whose only link was the `/<prefix>/` placeholder > - The final evidence read ran on a page that had not rendered the conversation yet, so it saved no links > - This pull request grades those cases on saved-state reads, the saved reply and resolved links, and reads final evidence from the rendered conversation > - The benefit is that these cases fail when the product is wrong and keep their links when it is right ## Linked Issues or Issue Description Refs #20998 Refs #20999 **What happened?** - `companies.agent-observability`: the board saved and tested a collector. Asked to acknowledge it, the agent said there was no endpoint or credentials. The oracle graded only human-side state, so it passed. - `inbox-search.show-my-work`: the saved reply said "Answered above" over nothing. The links the oracle counted came from run-stream prose that the bubble also shows. - `instance-operations.restart-service`: the reply's only action link was `/<prefix>/company/settings/instance`. The oracle did not check links. - Eight cohort-3 passes (`access.other-company`, `agents.agent-api-key`, `assistant-connections.revoke-personal-grant`, `budgets-costs.resolve-budget`, `recovery.retry-exhausted`, `steering.stop-versus-pause`, `task-coordination.pause-tree`, `teams.install-team`) saved a final observation with no links and a failed grade. The graded reload had the links. The final evidence read ran after `verifyResult` left the page mid-load, and an expired render wait from an earlier document made the read instant. **Expected behavior** Each oracle fails the observed wrong behavior and passes valid, reworded behavior. Final evidence holds the links that the conversation renders. **Steps to reproduce** Grade the retained cohort-3 observations with the old oracles. They pass. Grade them with these oracles. The three wrong cases fail. ## What Changed - `companies.agent-observability` (oracle v4): each acknowledgement turn must read the saved trace export status successfully. The read is bound to the company by its redaction-safe `call_api` receipt. A reply must also render a link to `/<prefix>/company/settings` for the fixture company. The reply's wording is not matched. - `inbox-search.show-my-work` (oracle v5): actionable links must be carried by the saved reply: a Markdown link target in the saved body, or a task identifier written in it, which the conversation links. New shared helper `savedReplyLinks`. - `instance-operations.restart-service` (oracle v3): new check `reply-links-resolve` rejects any rendered or saved link target that keeps a template placeholder (`<...>` or `{...}`). New helpers `markdownLinkTargets` and `unresolvedLinkTarget` in `chat-rendered-links.ts`. - Link capture: the final success-path evidence read reopens the rendered conversation first. An expired render wait now belongs to its loaded document, not to the Playwright page. - Updated `chat-catalog-descriptors.json` for the three oracle versions and source files. No scenario yaml changed, so the TestOps manifest does not change. ## Verification - `npx vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/chat-catalog-domains/identity-observability.test.ts tests/runner-e2e/chat-catalog-domains/internal.test.ts tests/runner-e2e/chat-catalog-domains/privileged-oracles.test.ts tests/runner-e2e/chat-catalog-domains/privileged-restart-history.test.ts tests/runner-e2e/chat-rendered-links.test.ts tests/runner-e2e/chat-catalog-consistency.test.ts tests/runner-e2e/chat-catalog-fingerprint.test.ts` - Calibration tests for each case: a valid result, a reworded valid result, and the broken cohort-3 behavior. - Red/green: the new render-wait test fails on the old wait logic and passes with this change. - `npx tsc -p tests/runner-e2e/tsconfig.json --noEmit` passes. `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check` reports no change. - A paid run on a combined branch with the two product fixes is recorded in a comment on this pull request. ## Risks - `companies.agent-observability` needs the agent-readable status route from #20998. Without it, the case fails, which is correct. - The show-my-work check needs links in the saved reply. Before #20999, an OpenCode reply that left its answer in the run stream fails this case. - The final evidence read can wait up to 30 seconds to reopen the conversation. The read is still bounded. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
2a1620f143 |
fix(heartbeat): deliver a person's answer that waited behind a turn that closed the task (#20996)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - An agent can ask the person a question on its task, and the person
answers it in the task chat
> - The person can answer while the agent's turn is still running. The
answer then waits for that turn to finish
> - If that turn completes the task, the release step cancels every
waiting wake for the assignee on the done task. The person's own queued
messages reopen the task, but the answer was dropped and no turn
received it
> - A person who answers an agent's question expects the agent to get
the answer. Losing it without a trace is a silent failure
> - This pull request treats an answer saved before the task was
completed as owed input. It reopens the task and delivers the answer
once
> - The benefit is that a person's answer is never dropped because the
agent finished its turn first
## Linked Issues or Issue Description
**What happened?**
The paid chat eval `steering.approval-delivery` showed it. The person
answered the agent's naming question while the agent's turn was running.
The turn completed the task. The answer's own wake was then cancelled
with "Deferred execution wake no longer applies to a terminal task". No
turn received the answer.
**Steps to reproduce**
1. Start a turn for the assignee on a task.
2. While the turn runs, answer the agent's question on that task.
3. Let the turn mark the task done.
4. The answer's wake is cancelled, and no run receives it.
**Expected behavior**
The answer reaches the assignee exactly once. An answer saved after the
task was completed is still dropped, as before.
Refs #20981, #19331
## What Changed
- `wake-queue/application/use-cases.ts`: before cancelling a deferred
assignee wake on a done task, check whether it carries a person's owed
answer. If it does, reopen the task to `in_progress` and promote the
wake. An answer continues the assignee's own execution, so the existing
continuation status gate applies unchanged.
- `wake-queue/adapters/postgres.ts`: add `owedInteractionAnswer`. It
applies only to an undelivered response that a person resolved, that is
not a tool action, and that was saved at or before the task's
`completedAt`. It returns `blocked` when unresolved blockers forbid
`in_progress`. `reopenIssue` takes an optional target status.
- The blocked case cancels the answer wake with an explicit reason and
does not reopen the task, so the release step does not fail. A
regression test reproduces the earlier throw (fix for the Greptile P1).
- `heartbeat-answer-behind-settling-run.test.ts`: an end-to-end
heartbeat test. It is red on master (the answer wake is cancelled and no
run is bound to it) and green here (one run is bound and succeeds). A
control test shows that an answer saved after completion is still
dropped.
- `use-cases.test.ts`: unit cases for an owed answer, a not-owed answer,
a cancelled task, and an agent-requested wake.
- Eval `steering.approval-delivery` oracle v7: also accept the Acorn
document when the worker's settling turn saved it after the answer was
saved. The eval still requires exactly one later turn bound to the
answer. Calibration tests cover the retained shape, reworded text, a
pre-answer save, the wrong name, another agent, a stale revision, and a
missing bound turn.
## Verification
- `cd server && npx vitest run
src/__tests__/heartbeat-answer-behind-settling-run.test.ts
src/modules/wake-queue
src/__tests__/heartbeat-blocked-question-answer.test.ts` (351 passed)
- Red check: with the `server/src/modules` changes reverted, the new
delivery test fails.
- `cd server && npx vitest run
src/__tests__/heartbeat-comment-wake-batching.test.ts
src/__tests__/question-response-delivery.test.ts` passed. In
`issue-queued-comments-routes.test.ts` all 138 tests passed, but a
cleanup hook timed out locally. CI covers this file.
- `npx vitest run --config tests/runner-e2e/vitest.config.ts
tests/runner-e2e/chat-catalog-domains/runtime-approval-queue.test.ts
tests/runner-e2e/chat-catalog-consistency.test.ts` (58 passed)
- `tsc --noEmit` passes for `server` and `tests/runner-e2e`.
- Paid single-case run 43136466619 on
|
||
|
|
8818402e7a |
feat(inbox): route requests to people and record personal outcomes (#20959)
## Thinking Path > - Paperclip helps people manage AI agents and their work. > - The inbox collects requests that need a human decision. > - Teams need to know who owns each request and who may act on it. > - A shared close operation must differ from hiding a card for one person. > - This pull request adds personal routing and durable outcomes to the existing attention service. > - People can claim or transfer work and inspect what another person did for them. > - Domain permissions still decide who may resolve each request. ## Linked Issues or Issue Description **Subsystem affected** Server, shared contracts, and database. This is the routing foundation for a stacked inbox UI change. **Problem or motivation** A company-wide attention feed does not give teams a clear personal queue. Archive, withdrawal, transfer, and decisions by another person need distinct scopes and records. **Proposed solution** Resolve a request audience from its addressee, task participants, and configured owners. Provide personal, unassigned, company, all-assignee, and archived views. Add claim, transfer, personal archive and snooze, shared withdrawal, receipts, and personal history. **Alternatives considered** A single shared inbox leaves ownership and personal hiding ambiguous. The service keeps one canonical request and projects its audience instead of copying requests for each person. **Roadmap alignment** This extends the existing Multiple Human Users and Agent Reviews and Approvals capabilities in ROADMAP.md. **Additional context** Refs #19363 and #19520 for the current inbox navigation and confirmation replacement rules. Refs #20268 for related actionable-request work. That PR filters actionability; this stack adds audience routing and personal outcomes. This is part 1 of a two-PR stack. The inbox UI PR #20961 targets this branch. ## What Changed - Add nullable company and project decision owners and a company budget owner. - Add audience views and personal badge counts to the canonical attention projections. - Add claim, transfer, withdrawal, personal archive, snooze, and restore operations with permission checks and activity records. Lock addressed transfers to the current addressee or routing administrators, and validate the recipient against canonical resolution policies. - Add receipts and scoped history for work handled by another person. - Keep closed-task requests and replaced confirmations out of current cards. - Add an idempotent migration and a concurrent receipt-history index. - Update typed fixtures so the existing UI remains compatible with the new contracts. ## Verification - Server layered typechecking passes, including all three test-tree partitions. - Seven routing and interaction-resolution suites pass all 249 tests against a dedicated external PostgreSQL test database on `4caecc21d8`. Six new regressions failed before the repair; they cover takeover followed by approval, legitimate delegation, excluded creators/review requesters, and unauthorized claims. - Prior focused verification covers private approval history and receipts, transfer visibility, audience escalation, withdrawal governance, hidden-card counts, per-recipient confirmation replacement, and SDK source contracts. - A synthetic three-person company passed 37 routing and lifecycle acceptance checks. The UI stack adds browser acceptance for the shared controls. - The complete [GitHub Actions CI matrix](https://github.com/paperclipai/paperclip/actions/runs/43134997331) passes on `4caecc21d891677323a7335f23b1a015cce24c13` (Linux/Node 24: typecheck, build, general and serialized tests, Runner support, browser shards, and aggregate gates). Greptile is 5/5 on that exact head with no open findings. The fresh security scan passes after explicitly rechecking the addressee-transfer repair. - `pnpm verify` reports exit 2 because a CI-global script changed; full CI supplies that broader coverage. Embedded PostgreSQL could not start on this Mac because shared-memory slots were exhausted, so local database checks use a separate external test database. Linux CI covers the embedded setup. ## Risks - Personal badge counts change to the viewer's assigned requests. Accessible feed totals and task membership keep their existing scope. - The migration adds nullable owner fields and a dismissal fingerprint. Its index runs outside a transaction. The migration is classified and rerunnable. - Routing changes the addressee used by resolution authorization. The current addressee or a routing administrator must authorize an addressed transfer; the recipient must satisfy writer access, canonical resolver exclusions, and task review policy under row locks. - The shared validator export barrel crosses 1,000 lines because it exports the new attention schemas. Schema implementation stays in the dedicated attention validator module. ## Model Used OpenAI GPT-6 through Codex. Used repository tools, code execution, and browser checks. This session does not expose a more specific runtime model ID or its context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal Paperclip ticket id - [x] I have run focused tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes (OpenAPI contracts and docs/api/attention.md) - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Scott Tong <scott@scottsmbpm5max.lan> Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
9dfdeaad86 |
Refine agent workspaces and organization charts (#20973)
Refine agent details, identity editing, responsive controls, and organization charts with three card tiers, clearer save feedback, and stable filtering. Guard instruction-file renames and root configuration updates in one transaction through the agent lifecycle integration, preserving conflict checks, rollback, and configuration history. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
|
42fd790e80 |
fix(evals): grade approval delivery on the saved answer, not on which queued wake is listed (#20981)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat eval catalog runs real chat journeys against real agents and grades the saved product state > - The `steering.approval-delivery` case checks that a person's answer to an agent's question stays separate from editable queued messages and reaches the agent once > - The task composer lists one pending wake at a time. It shows editable ordinary messages before a saved card response, by design since #19331 > - When the chat lead relays the person's answer as its own task comment, the composer lists that relay. The saved answer waits on its own wake and is not listed > - The oracle required the answer and the relay in one listing. The product never produces that shape, so the case timed out every time the lead relayed > - This pull request grades the case on the saved answer and on its delivery, and accepts either valid listing > - The benefit is that the case measures the promise in its expected outcome and still fails on a lost, replayed, or editable answer ## Linked Issues or Issue Description **What happened?** The paid chat eval `steering.approval-delivery` (OpenCode, DeepSeek flash) did the answer, edit, reorder, and discard steps. It then timed out while it waited for the answer to show in the queue. **Steps to reproduce** Run the case. Let the chat lead relay the answer as its own task comment. After the person's two messages are discarded, the queue lists only the relay. The answer is saved as `answered` and keeps its own pending wake. **Expected behavior** The case passes when the answer stays saved and unchanged, cannot be edited through the queue, and reaches exactly one later agent turn. Refs #19331, #20899 ## What Changed - `runtime-approval-queue.ts`: add `approvalResponseListing`. After the discards, accept the answer listed alone and locked, or only the chat lead's relay listed. Reject any person message or an editable answer. - Keep the edit attempt through the queue route for both listings. Require it to be refused and the saved answer to stay unchanged. Check the locked card in the UI when the answer is listed. - Add `classifyApprovalSuccessors`. Grade delivery on saved run state: exactly one later worker turn bound to the answer, with the question ID and Acorn in its native input. Allow at most one other turn that carries only the lead's relay. Reject any other turn. - Accept the Acorn document from the bound turn or the relay turn. Still require the earlier work to be kept, all runs succeeded, the queue empty, and the task done. - Oracle `approval-delivery.v6`. Run bound 5 with a minimum of 4, to allow the one relay turn. - Calibration tests: the retained relay shape passes, the original listed shape passes, and these fail: no bound turn, two bound turns, a person message left behind the relay, an unexpected worker turn, two relay turns, a recorded listing that disagrees with the queue, and a listed answer whose controls were not seen as locked. No YAML changed. The expected outcome already promises retention and once-only delivery, not a queue listing. ## Verification - `npx vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/chat-catalog-domains/runtime-approval-queue.test.ts tests/runner-e2e/chat-catalog-consistency.test.ts` (46 passed) - `npx tsc -p tests/runner-e2e/tsconfig.json --noEmit` (clean) - `node tests/runner-e2e/source-sdk-public-inputs.mjs <sha> --check` (no change) - One paid single-case run of this branch. The result is added below in a comment. ## Risks - Low risk. The change is in eval code only. No product behavior changes. - The grader now allows one extra worker turn for the lead's relay. A turn that carries any other input still fails the case. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
4b24352712 |
test(restart-chaos): stall the plugin during reattachment (CR4) and at launch (CR5) (#20966)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart chaos harness (`tests/restart-chaos/`) injects faults
into a real Paperclip server and Daytona sandboxes, and grades each
task.
> - Class R cell CR3 stalls the Daytona plugin worker for 45 seconds in
the middle of a tool call. It found the failure that #20940 fixed.
> - #20945 fixed the same stall during reattachment and at launch, but
only unit tests cover those paths.
> - This pull request adds two cells: CR4 stalls the plugin right after
a K1 restart, while the run reattaches, and CR5 stalls it while the run
launches.
> - The benefit is that a wave now exercises the recovery and launch
paths, not only unit tests.
## Linked Issues or Issue Description
Refs #20909, #20940, #20945
**What happened?**
CR3 stalls the plugin only in the middle of a tool call. The
reattachment reads (saved state, identity marker, process check) and the
launch reads (runnerd launch call, identity read) had no chaos coverage
for a plugin stall.
**Expected behavior**
The harness can stall the plugin during reattachment and during launch,
and it grades the task as it grades CR3.
**Steps to reproduce**
1. Run a wave with `--faults CR4,CR5`. Before this change, the harness
rejects these fault kinds.
**Paperclip version or commit**
master at
|
||
|
|
66344b76ae |
fix(native): restart a bootstrap whose Runner exited before writing provider state on a resumed sandbox (#20972)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs execute through a Runner (`paperclip-runnerd`) inside a remote sandbox. A warm Daytona sandbox is reused by later runs. > - When a Runner exits before `session.open`, the run does a bounded same-run bootstrap retry. Durable evidence proves that no provider session and no provider event exist. > - The retry fails when the Runner wrote `runner-state.json` but exited before it wrote its provider state. Host preparation then calls the sandbox state "unavailable" and refuses with `runner_harness_state_mismatch: resumed_unavailable`. The task needs Retry. > - This pull request reads the Runner state on its own when the full check fails, but only for a proven bootstrap restart of the same run. A non-suspended Runner state of this run is then the run's own interrupted bootstrap, and the existing restart path moves it aside and claims a new session root. > - The benefit is that a Runner that exits before `session.open` on a reused sandbox recovers without a person. Saved state, a verified backup and every other mismatch keep their refusal. ## Linked Issues or Issue Description **What happened?** Restart-chaos wave `w-20261111T230537-s6402` (Claude through ACPX, Daytona warm sandboxes, `--ephemeral-server`, K1/K2/K15 with `--k15-fresh-home`, seed 6402) needed Retry on 4 of 6 tasks, with the same signature after each restart kind. The first attempt of each run reused the warm sandbox. Its Runner failed with `runner_ingress_bind_conflict: failed to bind listener: Address already in use` and exited with `lifecycle=connecting`. Recovery logged "Restarting an incomplete native runner bootstrap on the same heartbeat run". The second attempt reused the environment and failed with `runner_harness_state_mismatch: resumed_unavailable`. The earlier wave `w-20261111T212429-s6402` had one run with the same error code. **Root cause** `inspectRemoteHarnessState` in `server/src/services/native-runtime/native-session-executor.ts` checks the persistence directories, `runner-state.json` and the provider state file in one shell test. A Runner that exits at bind time wrote `runner-state.json` but no provider state, so the one test fails and the result is `incompleteReason: "unavailable"` before the Runner state is read. The restart path (`restartableBootstrapState`) accepts only `not_suspended` and `provider_unopened`, so it is skipped. `claimUntouchedRemoteSession` then cannot create the session root that the first attempt created, and the resumed branch throws `resumed_unavailable`. On a created lease the same state fell through without a restart. **Expected behavior** A proven, interrupted bootstrap of the same run restarts on the same sandbox. The run finishes without Retry. **Steps to reproduce** Run the s6402 cell above, or add `--runner-bind-fault` (this pull request) to force the Runner exit on every run's first launch. ## What Changed - `server/src/services/native-runtime/native-session-executor.ts`: when the full harness check fails during host preparation, `inspectOwnUnpreparedBootstrap` reads `runner-state.json` alone. It returns `not_suspended` for this run only when all of these are true: this run has a proven bootstrap restart (`bootstrap_incomplete` recovery plus the stop proof), no failover backup exists, no archived rotation snapshot is in use, and the Runner state names this run, this Runner instance and this session with a lifecycle other than `suspended`. Otherwise the result stays `unavailable`, as before. The existing restart path then moves the session root aside (it is kept, not deleted) and claims a new one. - Chaos coverage: `--runner-bind-fault` in `tests/restart-chaos` gives every server of the wave `PAPERCLIP_TEST_RUNNER_BIND_FAULT=first_launch`. With `NODE_ENV=test` only, the remote Runner launcher starts a short-lived listener on the Runner's listen port before a Runner's first launch on that server, so the Runner exits with a real `runner_ingress_bind_conflict`. Documented in `tests/restart-chaos/README.md`. ## Verification - Red on master: `pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/native-session-executor.test.ts -t "bootstraps only an untouched provider session"` failed 2 of the new cells. Resumed lease: `Error: runner_harness_state_mismatch: resumed_unavailable` (the staging error). Created lease: no restart happened (`expected [] to have a length of 1 but got +0`). - Green: the same cells pass. The negative cells keep `resumed_unavailable`: no restart proof, a Runner state of another run, a `suspended` Runner state with missing provider state, and a failover backup present. - `native-session-executor.test.ts`: 1406 of 1406 pass. A new test proves the bind fault holds the listen port (a second bind gets `EADDRINUSE`) and fires once per Runner instance. - `tests/restart-chaos`: `runner-bind-fault.test.ts`, `question-handoff.test.ts`, `shape-cells.test.ts` pass. Server and harness `tsc --noEmit` pass. - Chaos wave: see the comment below for the before/after counts. ## Risks - The new read runs only for a proven bootstrap restart of the same run with a failed full check, so the normal path has no extra command and no wait before `turn.start`. - A partly saved session (`suspended` Runner state without provider state), a Runner state of another run and any failover backup keep today's refusal. The moved-aside root is kept for inspection. - A Runner that started under an older server and is restarted by this version takes the same path, because it reads only files that older Runners already write. - The bind fault is test instrumentation. It is inert unless `NODE_ENV=test` and the variable are both set. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, file edit, test runs). Default context window, extended reasoning on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Related: #20928 (restart a Runner that exited before session.open), #20874, #20863, #20953. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
afcf34c9da |
fix(evals): form repair-cost-recording's saved-usage debt and grade it by saved state (#20895)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat feature evals check that the chat agent helps a person with real product state > - The `budgets-costs.repair-cost-recording` case checks that the agent sends the person to the saved-cost retry control and then reports the true accounting result > - The grader checked the agent's replies with keyword regular expressions, and the scripted person gave the agent the harness's own accounting readback > - Keyword checks reject valid wording and accept wrong answers, so the case did not measure the promise > - The setup also never formed: the prerequisite run always settled, so no "Ready to record" saved usage existed > - This pull request forms that state with a temporary trigger on the case-owned database (no product change), and replaces the two keyword checks with checks of saved state, the agent's own product calls, and rendered links > - The benefit is a case that can run, and a grader that does not depend on wording and that still fails the relevant broken behavior ## Linked Issues or Issue Description **What happened** The case `budgets-costs.repair-cost-recording` graded the chat replies with keyword regular expressions (`honest-post-retry-accounting-result` and `guides-accounting-without-rerun`). The scripted follow-up message also told the agent the harness's accounting result. Separately, the setup never formed in paid runs: the prerequisite native run settles as an unpriced cost event (`missing_price`), not as a `retryable` saved-usage row. Accounting retries every 5 seconds, so a retryable row only exists while accounting keeps failing. **Expected behavior** The grader accepts any wording. It requires that the guidance before the retry links the Costs page, that the post-retry run reads saved cost data before it answers, and that the agent does not write cost, budget or accounting records or start work. **Steps to reproduce** 1. Read `gradeAccounting` in `tests/runner-e2e/chat-catalog-domains/identity-accounting.ts` on master. 2. Reword a valid reply (for example "Head to Costs and press the button there; the work stays as it is."). 3. See that the keyword checks reject it. Refs #20600 ## What Changed - New `tests/runner-e2e/chat-accounting-fault.ts`, following `chat-archive-storage-fault.ts`: a temporary trigger on the case-owned database refuses cost-event writes for the one prerequisite task. The product's own accounting transaction rolls back, so the finished run keeps its saved usage pending and the product's automatic retries keep failing. Refused writes are counted in a rollback-proof sequence. It is allowed only when this case is the single selected execution, and it never writes product rows. - Setup: the case creates the prerequisite task unassigned, arms the trigger, assigns the task, and waits until Costs shows the genuine `retryable` row. The trigger is closed on a setup failure and at cleanup. - Act: the trigger is dropped right before the person clicks "Retry cost recording". Automatic recovery can record it first; the saved accounting read afterwards classifies either result (an unpriced event still shows in data quality, so it counts as unresolved). - `chat-archive-storage-fault.ts`: export its owned-database resolver for reuse (no behavior change). - `identity-accounting.ts`: remove the reply keyword checks for the repair case. - Add `initial-guidance-links-saved-cost-retry`: a reply from before the retry has a rendered link to the Costs page (`/<prefix>/activity/costs` or the `/<prefix>/costs` redirect). - Add `post-retry-reply-reads-saved-accounting`: the post-retry chat run made a successful read of a company cost report, or of the cost summary of the prerequisite task (bound by its redacted input digest). - Add `no-agent-accounting-fabrication-or-rerun`: no successful chat-agent call writes cost, finance, budget or accounting records or starts run or task execution. A denied attempt is not a write. - `observe` collects the chat agent's redacted `call_api` uses and results (`leadApiCalls`). - The scripted follow-up now reports only the toast that the page showed. That toast says "Costs recorded." even when the usage still has no price. - Own `oracleVersion` (`paperclip.chat-catalog.repair-cost-recording.structural.v3`) and source files in code and in `chat-catalog-descriptors.json`. - Calibration tests: reworded valid replies pass; a missing or late Costs link, a run link only, a foreign link, no fresh read, a read in the wrong run, a denied read, an unrelated read, another task's cost summary, and agent cost or execution writes fail. ## Verification - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/chat-accounting-fault.test.ts tests/runner-e2e/chat-archive-storage-fault.test.ts tests/runner-e2e/chat-catalog-domains/identity-accounting.test.ts tests/runner-e2e/chat-catalog-contract.test.ts tests/runner-e2e/chat-catalog-consistency.test.ts`: all pass. The embedded-Postgres test drives the product's `accountRunCost`, `reconcileRunCosts`, data-quality report and board retry against the trigger: only the faulted task's run stays `retryable`, other debt in the company settles, and the retry records it once the trigger is dropped. - `npx tsc --noEmit -p tests/runner-e2e`: no errors in the changed files. - `node tests/runner-e2e/source-sdk-public-inputs.mjs <sha> --check`: unchanged. - Paid run still needed: `chat-features.runner-opencode.local.budgets-costs.repair-cost-recording`. ## Risks - Low risk. The change affects only this eval case. - The new read check needs the chat agent to have company cost-read access. Agents cannot read `costs/data-quality` (board only), so the agent uses the cost reports or the task cost summary. - The trigger runs DDL on the case-owned database only; the same isolation checks as the archive storage fault apply, and it is dropped before the retry and removed at cleanup. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
a75d7df9c7 |
fix(evals): gate steering approval-delivery and steer-active on real product state (#20899)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The chat feature catalog runs real native agents to prove the chat and task-steering promises. > - Two steering cases failed in setup before they could measure the product. > - `steering.approval-delivery` waited for a worker run to stay running while its question was pending. The product parks a native turn as soon as a durable question is saved, so that state never occurs. > - `steering.steer-active` waited for the queue to say steering is "available". The OpenCode harness reports that it cannot steer, so the queue says "unsupported". > - This pull request gates each case on the real product state and grades the behavior that state promises. > - The benefit is that both cases now measure their user promise on the evaluated profile, without faked runs or faked steering. ## Linked Issues or Issue Description **What happened** Both cases failed during setup on `runner-opencode`. `steering.approval-delivery` stopped with "The actual question fixture did not reach its running pending-input boundary": the worker's `request_human_input` call parked its turn (run succeeded, `reportedWorkDisposition: yielded`, continuation `response_wake` for the question). `steering.steer-active` timed out waiting for `steeringDisposition: "available"`; the OpenCode driver declares `steering: false`, so the queue reports `"unsupported"` and the page shows Interrupt instead of Steer. **Expected behavior** `steering.approval-delivery` should treat the saved yield and the pending question as the pending-input state, queue ordinary comments and the bound answer behind a real active worker turn, and prove one continuation receives the answer. `steering.steer-active` should grade same-turn steering when the harness can steer, and grade the honest fallback when it cannot: the message stays visibly queued, nothing is recorded as steered, and one normal turn consumes it. **Steps to reproduce** 1. Dispatch `chat-features.runner-opencode.local.steering.approval-delivery` on master. 2. See the setup stop after the worker run succeeds with a yielded result. 3. Dispatch `chat-features.runner-opencode.local.steering.steer-active` on master. 4. See setup time out while `GET /api/issues/:id/queued-comments` reports `steeringDisposition: "unsupported"`. ## What Changed - `runtime-approval-queue.ts`: gate the question on the parked turn (`pendingInputYield`: succeeded, yielded, `response_wake` keyed to the question). An ordinary human message then wakes a real second Morgan turn that waits at the task gate. Ordinary comments and the bound answer queue behind that turn. The run count goes from 3 to 4. - `runtime-approval-queue.ts`: remove the reply keyword checks (`acornNamingDirection` and the "I have answered" regex). Grade exactly one saved answer path instead: the human response control, or an attributed canonical response. - `runtime-active-steering.ts`: accept either definitive capability. Unsupported branch: no enabled Steer control, a visible Interrupt control, a steer attempt with the person's session refused with `409 steering_unsupported`, no `issue.queued_comment_steered` activity, the exact message still queued, and one continuation that consumes it and registers a rendered blue `website.html`. - `runtime-active-steering.ts`: remove the reply keyword checks and the `/blue/` match on the delivery document. The rendered page proves the color. - Update calibration tests in both directions, the workspace setup test, and the descriptors (`oracleVersion` bumps, `expectedRunCount` 4 for approval-delivery). The scenario YAML is unchanged. ## Verification - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/chat-catalog-domains/runtime-active-steering.test.ts tests/runner-e2e/chat-catalog-domains/runtime-approval-queue.test.ts tests/runner-e2e/chat-catalog-domains/runtime-approval-workspace.test.ts tests/runner-e2e/chat-catalog-contract.test.ts tests/runner-e2e/chat-catalog-consistency.test.ts`: 87 passed. - `npx tsc --noEmit -p tests/runner-e2e`: no errors in the changed files. - `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`: unchanged. - The live proof is a paid run of `chat-features.runner-opencode.local.steering.approval-delivery` and `chat-features.runner-opencode.local.steering.steer-active`. ## Risks - approval-delivery now depends on a human message waking the worker while its question is pending. The product does this (`supersedeOnUserComment: false` keeps the card open, and comments wake the assignee unless the task is done or cancelled). If the worker does not run the gate on that turn, setup fails with a named label. - steer-active on OpenCode now measures the honest unsupported fallback, not same-turn delivery. A steering-capable profile still runs the same-turn branch. - Low risk to the product: only eval code changes. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
ba5ba54439 |
fix(evals): name unmet completion-gate readiness fields and grade the gate read, not reply words (#20887)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat catalog eval `tasks.completion-with-gate` checks that the CEO does not force Done while an independent human review is still required > - Its setup waits for a real Morgan implementation that awaits the configured review stage > - The setup timed out and did not say which state field was wrong > - The retained evidence shows a product defect: the native completion path never started the configured review stage (fixed in #20882) > - This pull request makes the readiness timeout name each unmet field, accepts valid product continuation runs, and replaces reply keyword grading with a structural check > - The benefit is that the eval measures the gate, and its setup failures are self-explaining ## Linked Issues or Issue Description **What happened** The eval failed with "Timed out waiting for Real prior Morgan implementation awaiting independent review". Morgan uploaded `website.html` and reported needs_review. The task became in_review through a generic review card, but `executionState` stayed null, so the poll (pending stage for Riley) never matched. The run finished at 229 s of the 240 s setup window. The grader also matched reply words with regular expressions. **Expected behavior** Setup reaches the configured review stage through supported product paths. A timeout names each unmet field. The grader proves the outcome from saved state and run receipts, not reply wording. **Steps to reproduce** 1. Dispatch `chat-features.runner-opencode.local.tasks.completion-with-gate` on master. 2. Read `server.log` and `snapshots/chat-final-run-ledger.json`: `authoritativeDecision: in_review`, `finalizationReasonCode: actionable_attention_pending`, no executionState. ## What Changed - `internal-task-completion.ts`: `unmetPriorDeliveryReview` lists each unmet readiness field. The poll uses it for `accept` and `timeoutDetail`. - Setup and grader accept one or more succeeded Morgan runs (a product continuation is valid). Any failed, cancelled or timed-out worker run is an unmet readiness condition, so setup stops before the paid CEO turn. The delivery must come from one of these runs. - The `truthful-outstanding-review` reply regex check is replaced by `checked-actual-completion-gate`: the CEO's chat run read the task (`GET /api/issues/{id}` or `get_task`, matched by record-bound receipt) and replied. The saved pending stage, unchanged policy and absence of a forced Done still prove the outcome. - Run bound 2 → 3, setup window 240 s → 330 s, attempt 540 s → 660 s. `oracleVersion` is `paperclip.chat-catalog.completion-with-gate.v2`. The descriptor is updated. - Cherry-picks both #20882 commits so the paid run exercises the product fix. ## Verification - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts` on `internal.test.ts`, `internal-comment-reference.test.ts`, `internal-edit-ownership.test.ts`, `chat-catalog-contract.test.ts`, `chat-catalog-consistency.test.ts` (184 tests) and 12 other suites that reference the case (133 tests). - Calibration: two reworded valid replies pass. A missing gate read, an empty reply, a forced Done and a failed worker run fail. The retained product state (in_review, no executionState) produces named unmet fields. - `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`: unchanged. - `npx tsc --noEmit -p tests/runner-e2e`: no new errors in the changed test files. The local worktree has unbuilt packages, so CI supplies the full typecheck. ## Risks - The reply's meaning is no longer machine-checked. A person must read the reply in the paid run. - The larger run bound and windows accept slower valid runs. Readiness still fails on any failed worker run. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
23909077ad |
fix(retry): re-deliver the failed run's wake when the board retries it (#20970)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - When a run fails, the board shows a Retry button. The Retry starts a
new run with the wake reason `task_retry`.
> - A failed run can serve a wake that happens once, for example a
scheduled task monitor check (`issue_monitor_due`). The monitor is
consumed when it fires.
> - The Retry run did not tell the agent which wake it retries. The
agent saw only `task_retry` and the earlier interruption.
> - In the restart-chaos harness, the agent followed its brief ("act
only on a board comment or a scheduled monitor check"), decided that the
Retry was not the monitor check, and changed nothing. The run succeeded,
but the task stayed `in_progress` with no run, no wake and no monitor.
The person clicked a Retry that worked mechanically but lost the work it
was for.
> - This pull request records the retried run's own wake reason on the
Retry's continuation trigger (`trigger.retriedReason`) and shows it in
the wake summary.
> - The benefit is that a Retry re-delivers the same wake that failed.
Dotta's rule: if a Retry button shows up, it still needs to work.
## Linked Issues or Issue Description
**What happened?**
Restart-chaos wave `w-20261111T230537-s6402` (Claude, Daytona warm,
volume-free server mode, restart fault K1), task t03 (workload W2). The
phase-2 monitor wake started run `60d821af`. That run failed with
`native_session_interrupted`. The board Retry was accepted (HTTP 202).
Retry run `a5a3958e` started 2 s later and succeeded. Its context had
`wakeReason: task_retry` and `executionContinuation.trigger = { reason:
"task_retry", sourceRunId: "60d821af…" }`, with no reference to
`issue_monitor_due`. The agent's summary: "This wake was a task_retry
resuming the previously interrupted run, not a new board comment or
scheduled monitor check, so per instructions I changed nothing." The
issue stayed `in_progress` with `executionRunId = null`, monitor status
`triggered` and no `nextCheckAt`. The harness graded
`R3:retry_did_not_complete` and U1 `no_owner_no_action 182s`.
**Expected behavior**
A Retry of a failed run re-delivers that run's wake. A Retry of a failed
monitor check is handled as that monitor check.
**Steps to reproduce**
1. Give an agent a task that acts only on a scheduled monitor check.
2. Let the monitor fire, and make the monitor run fail (restart fault K1
during the run).
3. Click Retry on the task.
4. Before this change, the Retry run's prompt says only `reason:
task_retry`, and the agent does not do the monitor work.
**Additional context**
The same signature (`R3:retry_did_not_complete:W2C:K0`, Codex, monitor
rewake run `bac90594` failed `native_session_interrupted`, Retry run
`cfb24146` succeeded without the phase-2 marker) is in wave
`w-20261111T192658-s5201` task t06. That case was attributed to the
response-wake loop that #20935 fixed. Its shape matches this cause: the
Retry run served the monitor wake but did not do the monitor work. That
wave kept no database, so I could not read its run context directly.
Refs #20935
## What Changed
- `buildExecutionContinuation` adds `trigger.retriedReason` when the
current wake is a Retry (`task_retry` or `retry_failed_run`). The value
is the source run's own wake reason. A Retry of a Retry inherits the
first non-retry reason from the source run's saved continuation.
- `ExecutionContinuationEnvelope.trigger` gets the optional
`retriedReason` field.
- `renderPaperclipWakePrompt` adds `- retried wake: <reason> (this Retry
re-delivers the wake the failed run did not finish; handle it as that
wake)` to the wake summary for fresh and resumed sessions.
- New test in `server/src/services/execution-continuation.test.ts` on
embedded Postgres: a failed `issue_monitor_due` run, a Retry of it, a
Retry of that Retry, and a non-retry wake.
## Verification
- Red on master: `cd server && npx vitest run
src/services/execution-continuation.test.ts -t "carries the failed run"`
failed with `trigger` = `{ reason: "task_retry", sourceRunId }` and no
`retriedReason`.
- Green: the same file passes (45 tests).
- `tsc --noEmit` passes for `packages/shared`, `packages/adapter-utils`
and `server`.
- `pnpm verify` exits 2 (a shared type changed), so required CI supplies
broad coverage.
- Chaos: the existing cell W2 × K1 (Claude, Daytona warm) reproduces
this when the fault lands in the monitor rewake run. W2C (Codex) shows
the same shape. I did not rerun a wave for this PR.
## Risks
- Low risk. The change adds one optional field and one prompt line. It
changes no admission, lock, status or authority path.
- The field is computed only for Retry wakes. Other wakes are unchanged.
- Runs saved by an older version have no `retriedReason`. A Retry built
from them reads the source run's `wakeReason`, which older runs already
saved.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, local test execution).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
ba905c7254 |
fix(evals): name each team member's readiness and grade configure-team structurally (#20879)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat catalog eval `teams.configure-team` checks that chat gets an installed team ready member by member > - Its setup installs Product Engineering, removes the QA member's account, and requires a real probe split: QA missing an account, the senior coder ready > - The QA probe returned a generic `opencode_hello_probe_failed` error, so the setup stopped with "probes do not establish the required missing-versus-ready account split" > - The cause is a product classification gap: OpenRouter answers an empty key with HTTP 401 "Missing Authentication header", which the OpenCode probe did not treat as authentication (fixed in #20878, cherry-picked here) > - This pull request makes the setup blocker name each member's observed readiness, and replaces reply keyword checks with structural checks > - The benefit is that the eval can reach the chat journey and grades it without matching words ## Linked Issues or Issue Description Refs #20878 **What happened** `teams.configure-team` failed in setup. The QA member's probe was `fail` with `opencode_hello_probe_failed`. The senior coder passed. The setup error did not say which member was in which state. **Expected behavior** The QA member's probe reports the missing-authentication check, so the split forms and the chat journey runs. When the split does not form, the setup error names each member's observed status and checks. **Steps to reproduce** 1. Run `chat-features.runner-opencode.local.teams.configure-team` on master. 2. Read `snapshots/chat-catalog-team-initial-probes.json`. 3. QA shows `fail` with `opencode_hello_probe_failed:error`; the setup stops. ## What Changed - Cherry-pick of #20878: the OpenCode probe classifies a provider 401 as `opencode_hello_probe_auth_required`. - The setup blocker now lists each member as `name (expected missing account | expected ready | other member): status [non-info checks]`. - Grader: remove the member-name substring check and the "team is ready" regex. Identification is the rendered link to the missing member's own page, which the case already required. The followup is graded by a non-empty reply from its own succeeded run, not by the words "ready/configured/verified/account". Fresh per-member probes, preserved working configuration and revisions, and the attributed repair stay unchanged. - Calibration: reworded and non-English valid followups pass; missing handoff, empty followup, and failed followup run fail. A setup test with the recorded failure shape checks the new error text. - `oracleVersion` bumped to `paperclip.chat-catalog.team-configure.v3`; `parse.ts` added to `sourceFiles`. ## Verification - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/chat-catalog-domains/external-team-configure.test.ts tests/runner-e2e/chat-catalog-domains/external-team-configure-prerequisite.test.ts tests/runner-e2e/chat-runtime-profile.test.ts tests/runner-e2e/chat-catalog-contract.test.ts tests/runner-e2e/chat-catalog-consistency.test.ts` passes. - `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check` reports no change. - `node --test tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs` passes. - The paid run `chat-features.runner-opencode.local.teams.configure-team` is still to run. ## Risks - The removed "team is ready" regex failed open. A false whole-team claim in the reply now needs human reading of the transcript. The structural checks still reject a missing handoff, a missing repair, and any member that fails its fresh probe. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
b7446969d7 |
fix(evals): give choose-capable-target a saveable computer and grade it structurally (#20881)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat catalog evals check that agents answer real user questions through the product. > - The case `execution-environments.choose-capable-target` asks "Which computer can run Morgan with Claude Code?" while environment reads are board-only. > - Its setup saved a built-in fake sandbox as the "unsupported" computer. The product refuses to save fake sandboxes (since #4449), so setup failed with 422 before any agent run, and the test API hid the reason. > - No computer that the product can save is advertised as unable to run Claude Code, so the original contrast cannot be built through supported APIs. > - This pull request changes the contrast to advertised support versus runtime health, grades the reply by structure instead of keywords, and makes setup failures show the product's validation reason. > - The benefit is that the case can measure its user promise, and a future setup 422 says why. ## Linked Issues or Issue Description **What happened** In the full chat catalog run, setup failed with `POST /api/companies/<id>/environments returned 422; body withheld`. A focused server test shows the reason: "Built-in fake sandbox environments are reserved for internal probes and cannot be saved." The fixture payload was `driver: "sandbox", config: { provider: "fake" }`. **Expected behavior** Setup builds its computers only through supported APIs, the agent run starts, and the grader checks the user promise without reply keywords. A setup validation failure states the product's reason. **Steps to reproduce** 1. Run `chat-features.runner-opencode.local.execution-environments.choose-capable-target` on master. 2. Setup posts a fake sandbox environment and gets 422. 3. `cd server && pnpm exec vitest run src/__tests__/environment-routes.test.ts -t "rejects persisted fake sandbox environments"` shows the same rule. ## What Changed - `runtime-capability-choice.ts`: setup saves an SSH computer on a reserved `.invalid` host (no key, never probed) beside the local fixture computer. Production capabilities advertise Claude Code on both. The `enableEnvironments` experimental setting is still set only through `PATCH /api/instance/settings/experimental` and restored in cleanup. - Grader: removed the reply regex and keyword checks. New checks: agent gets 403 on capabilities and probe; the reply renders a link to `/<prefix>/company/settings/instance/environments` (or a target's edit page); the reply has no board-only SSH settings (host, user, path); targets (including status and metadata) and the worker are unchanged; exactly one chat turn and no tasks. `verifyResult` clicks the rendered link and checks that the page shows both computers and loads the same capabilities. - `execution-environments.yaml`: context changed from "one target supports the harness and another does not" to support versus health, with the reason. Recorded in `chat-full-catalog-testops-manifest.json` `changedFields`, pinned in the manifest test, and the inventory digest is updated. - `oracleVersion` is now `paperclip.chat-catalog.runtime-capability-choice.v2` in code and in `chat-catalog-descriptors.json`; `sourceFiles` adds `internal-support.ts`, `server/src/routes/authz.ts` and `server/src/services/environment-config.ts`. - `api.ts`: cookie-session 400/409/422 failures now include the product's `error` message, at most 240 characters, with request string values (8 or more characters), the session cookie and long token-like strings redacted. `details` and all other statuses stay withheld. - Includes a cherry-pick of #20880 (agent environment denials link the environment settings page). Without it, the agent has no route to give. ## Verification - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts` on `runtime-capability-choice.test.ts`, `runtime-computers.test.ts`, `chat-catalog-contract.test.ts`, `chat-catalog-consistency.test.ts`, `api-private-transport.test.ts`, `api.test.ts`, `api-response-reading.test.ts`: 73 passed. - Grader calibration: valid and reworded replies with the settings link pass. Missing, unprefixed, wrong-prefix, cross-origin and wrong-page links fail. Replies with the SSH host, user or path fail. Unsupported capability, reachable host, 200 boundary, changed target or worker, an extra run and a created task fail. - `node --test tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs`: 10 passed. - `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`: unchanged. - `npx tsc --noEmit -p tests/runner-e2e`: no errors in changed files. - Paid run still needed: `chat-features.runner-opencode.local.execution-environments.choose-capable-target`. ## Risks - Premise change: the case no longer has a saved computer that is advertised as unable to run Claude Code, because the product cannot save one. The "avoid an incompatible target" clause stays in the expected outcome, but no saved target exercises it. - The reply's claims about health are not graded by words. A reviewer must read the paid run's reply. - The 400/409/422 reason can show product text that the test API hid before. It is bounded and redacted. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
c45f54ee74 |
fix(native): stop or release unclosable idle warm Runners on environment delete, and say why a close failed (#20957)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs keep a warm Runner in a Daytona sandbox between turns of
a task, so the next turn starts fast
> - Each idle warm Runner has an in-memory owner in the controller.
Deleting an environment closes these owners before it destroys the
sandboxes
> - In every warm chaos wave today, some owners never closed. The server
logged only `{"failed":2}`. Environment delete returned 409 forever, and
all sandboxes of the wave stayed `started` until their 60-minute TTL
> - On staging, that is leaked sandboxes (cost) and owners that block
the work after them
> - This pull request logs and reports the exact close error, stops an
unsettled idle Runner when its environment is deleted, and releases an
owner whose sandbox is already destroyed
> - The benefit is that environment delete finishes or names the exact
run and error, and every warm chaos wave now grades a clean shutdown
## Linked Issues or Issue Description
**What happened?**
After warm Daytona chaos waves (W2, W2Q, W2C cells, `--faults K0`) on
Codex, Claude and OpenCode, the server logged `idle native session
cleanup failed before controller shutdown {"idleSessions":{"failed":2}}`
with no reason. `DELETE
/api/environments/:id?destroyReusableSandboxLeases=true` returned 409
"warm native runner did not shut down cleanly" on every retry. Eight
sandboxes per wave stayed `started`.
**Root cause.**
`closeIdleWarmNativeSessions`
(`server/src/services/native-runtime/native-session-executor.ts`)
dropped the close error (`catch {}`) and counted it. The owners that
failed are the W2Q owners. A W2Q task parks a question and is answered
by a later run on a fresh sandbox. With the reason logged, a wave shows
two causes for the same owner:
1. The owner's close cannot settle (`native_session_close_unrecoverable:
runner did not durably suspend before checkpoint`, the governed-wait
shape from #20844). Only a successor reset could force a stop
(`discardUnsettledSession`). An idle sweep or environment delete had no
forced stop.
2. The owner's sandbox lease was already destroyed by the
pending-cleanup sweep after the successor moved to a fresh sandbox
(#20863). Every close then fails with `Daytona sandbox lease <id> is no
longer active.`, because the stop RPC has no sandbox to reach. The owner
stays registered for ever.
The Runner close memoizes its rejection, so every later attempt fails at
once with the same error. This is not the "controller sequence must be
contiguous" exit (#20946): no close failure in these waves names a
sequence mismatch.
**Expected behavior**
An idle warm owner with no turn in flight never blocks environment
delete for ever. Every failed close names its run and its error.
**Steps to reproduce**
1. Run `tests/restart-chaos/main.ts --env daytona --daytona-warm
--adapters claude --workloads W2,W2Q,W2C --faults K0 --concurrency 8
--tasks 8`.
2. Read `<wave>/sandbox-sweep.json`: `environmentDeleteBlocked` is set
and the sandboxes are `started`.
## What Changed
- `closeIdleWarmNativeSessions` now returns `failures: [{ runId,
issueId, environmentId, providerLeaseId, reason }]`. The reason is the
close error with its code, its settlement record and its cause. It also
logs `idle warm native sessions did not close` with that list. The
idle-timeout close logs its error too.
- `closeWarmNativeSessionsForEnvironment` takes `forceStopUnsettled` and
`db`. The environment delete route sets both. When the graceful close
fails, Core retires the owner as #20844 does: one confirmed Launcher
`stop`, the credential snapshot, `complete`, then instruction
collection. A failed forced stop is added to the reason.
- When retirement still fails and the owner's sandbox lease has a
`destroyed` termination receipt, Core releases the owner. It does not
contact the sandbox. It logs `warm native session's sandbox was already
destroyed; released its owner` with the close error.
- The environment delete 409 now names each run and its reasons, and
`details.nativeRunnerSessions.failures` lists them.
- Idle timeout, controller restart and tenant sleep do not force a stop.
Their conversation can still be resumed, so they only log the reason.
- Chaos harness: new check `warm-shutdown.ts`, "no leaked sandboxes /
clean warm shutdown". Every warm wave (`--daytona-warm` or `--lifecycle
warm`) runs it. It fails the wave on a blocked delete, on a sandbox left
running after cleanup, or on a warm session that did not close at
environment delete or controller shutdown. It writes
`<wave>/warm-shutdown.json` and a section in `SUMMARY.md`.
- Docs: `doc/plugins/PLUGIN_RUNNER_EXECUTION.md` and the chaos README.
## Verification
- RED on master: `native-session-executor.test.ts -t "stops an idle warm
owner whose close cannot settle"` fails: `expected { closed: 0, busy: 0,
failed: 1 } to match object { closed: 0, failed: 1, failures: [...] }`
(a bare count, no reason). `environment-routes.test.ts -t "names the
warm runner session"` fails: the 409 does not name the run. On the first
fix commit, `-t "sandbox lease is gone"` fails: the lease state is never
read, and the owner stays.
- GREEN: those tests pass. The executor suite, environment routes, run
control, recovery, heartbeat process recovery, instance settings, the
idle-sleep, eviction, release-resume, restart-recovery and
warm-checkpoint integration suites pass (11 files, 2471 tests).
`tests/restart-chaos` unit test `warm-shutdown` passes. Server and
harness `tsc --noEmit` pass.
- Daytona warm waves (Claude, W2/W2Q/W2C, K0, 8 tasks). The runner image
`sha256:e04d9ce4…` matches
|
||
|
|
068d814735 |
test(chaos): run waves in Cloud's volume-free server mode; make W2 phase 1 yield (#20930)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness (`tests/restart-chaos/`) runs real agent tasks against an isolated control plane and injects faults > - Cloud servers run in the volume-free mode (`PAPERCLIP_EPHEMERAL_SERVER=true`, durable agent files, run logs in object storage) > - The harness never set that mode, so every wave ran a different server mode from staging. Paths that exist only in that mode were never tested. One of them is the per-agent directory operation lock, so `--fail-recovery-step directory-lock` could never reach its step with any workload > - The re-wake workloads W2 and W2C also lost coverage: an agent that reported phase 1 with disposition `done` closed the task, and the harness never sent the re-wake (four of eight Claude cells on 2026-11-11) > - This pull request adds `--ephemeral-server` and makes the W2/W2C brief ask for a `yielded` phase-1 report > - The benefit is that chaos waves can run in staging's server mode and the re-wake cells test the re-wake on every provider ## Linked Issues or Issue Description **What is the problem?** The restart chaos harness cannot run the control plane in the volume-free mode that Cloud and staging use. The `directory-lock` recovery step (`agent_directory.operation_lock.before_acquire`) is reached only for disposable remote copies, which need `PAPERCLIP_EPHEMERAL_SERVER=true`. W2 and W2C cells also stop before the re-wake when the agent reports phase 1 as `done`. **What is the expected behavior?** A wave can opt in to the volume-free mode on Daytona, and every restart of the wave keeps that mode. The W2 and W2C cells reach the re-wake. **Steps to reproduce** Run `--fail-recovery-step agent_directory.operation_lock.before_acquire:1` on Daytona with any workload: the check `recovery_step_failed_once` reports "never reached". Run W2 on Claude: the first run ends with the task `done` and `rewake_sent` is "never sent". ## What Changed - `assertIsolatedServerEnvironment` (`tests/runner-e2e/harness-env.ts`) takes `expected.generatedSecretsMasterKey`. It admits `PAPERCLIP_SECRETS_MASTER_KEY` only when that parameter is set and equals the value exactly. `PAPERCLIP_SECRETS_MASTER_KEY_FILE` stays refused. The check runs over the final server environment. - `--k15-fresh-home` implies `--ephemeral-server` and defaults `--storage` to `s3`. A fresh instance home without the volume-free mode loses the agents' instruction bundles, which staging keeps in Postgres and object storage. - `--ephemeral-server` (`config.ts`, `types.ts`, `main.ts`, `supervisor.ts`, new `ephemeral-server.ts` with its unit test `ephemeral-server.test.ts`): it needs `--storage s3` and `--env daytona` and refuses K14. Every server of the wave gets `PAPERCLIP_EPHEMERAL_SERVER`, `PAPERCLIP_DURABLE_AGENT_FILES`, `PAPERCLIP_SHARED_UI_ASSETS`, the wave's Postgres as `PAPERCLIP_DATABASE_LISTEN_URL`, one stable `PAPERCLIP_SECRETS_MASTER_KEY` (also written to the config key file) and `RUN_LOG_S3_*` in the wave's object store. - `rewake.ts`: the W2/W2C phase-1 brief asks for `paperclip_finish` with disposition `yielded` and a `response_wake` continuation. - W2/W2C tasks are created in `backlog` with their assignee, and a board comment starts phase 1 (`startComment` in `workloads.ts`, used by `observer.ts`). A passive board wait needs a comment-started run; without it the yield queues an immediate continuation and phase 1 wakes itself. The start comment is posted once (saved-comment check plus a stable `clientRequestId`), and the brief keeps phase 2 out of blocking `remainingWork`. - Volume-free mode drops the repository workloads (B1, B5, F1, R1) from the default list and refuses them when listed: their fixture is a `local_path` project source, which a volume-free server refuses. - `README.md`: a section for the new flag and a note on the brief. - `shape-cells.test.ts`: option parsing and refusals for the new flag. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts tests/restart-chaos/shape-cells.test.ts` passes (16 tests). - `node scripts/typecheck-test-projects.mjs tests/restart-chaos` passes. - Live proof wave (Daytona, Codex, W2, K1 and K15 with a fresh home, `--storage s3 --ephemeral-server`): wave `w-20261111T205259-s6301` (2 tasks). The first server and both replacements (K15 with a fresh home, then K1) started and reached ready in the volume-free mode. The server process had `PAPERCLIP_EPHEMERAL_SERVER=true`, `PAPERCLIP_DURABLE_AGENT_FILES=true` and `RUN_LOG_S3_BUCKET` set. Both W2 tasks left phase 1 `in_progress` (yielded), received the re-wake and ended `done`. The wave also found product faults (reported separately, not fixed here): a re-wake after K1 failed with `runner_harness_state_mismatch: backup_without_reusable_lease` and needed Retry. - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/support.test.ts` passes, including the new master-key admission test (a matching wave key passes; an inherited key, a mismatched key and `PAPERCLIP_SECRETS_MASTER_KEY_FILE` are refused). ## Risks - Low risk. The flag is opt-in and the harness is test-only. The brief change alters the W2/W2C prompt, so results from older W2/W2C seeds are not directly comparable. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
ef755cecc6 |
fix(evals): grade case facts by the saved fact, bind case kickoffs to the CEO run, stop block-on-design on a missing blocker (#20948)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat eval catalog runs real chat journeys and grades saved
product state
> - Three cases failed in runs 43088288923 and 43101125481. In two of
them the product did the right thing
> - `cases.update-case-facts` used a word allow-list on a saved field.
`cases.create-case` read a field the product never fills for a new task
> - `task-coordination.block-on-design` timed out in the browser on a
missing row, which hid the real cause
> - This pull request grades what the user asked for, binds records by
structure, and stops with the real cause
> - The benefit is that each case fails only when the product fails, and
says why
## Linked Issues or Issue Description
**What happened**
- `cases.update-case-facts`: the CEO saved `browser: "Mobile Safari only
(bug reproduces only on mobile Safari)"` and kept the other fields. The
grader allowed only the words "safari" and mobile qualifiers, so "bug
reproduces" failed the first phase. The earlier rewording of the
scripted follow-up was not the cause: it is sent only after this phase.
- `cases.create-case`: the CEO created the case, the `workflow` and
`customer_brief` documents and Morgan's kickoff in one run, before
Morgan started. The grader required the kickoff's `sourceRunId` to be a
CEO run. The product sets `sourceRunId` to the run that holds the task
being handed off, so it is always null when the kickoff creates the
task. The grade never passed, so the harness kept waiting. Morgan then
found no app to test, handed the case back to the CEO, and the extra CEO
run broke the run bound.
- `task-coordination.block-on-design`: the CEO saved no Design blocker.
The scripted human then waited 30 s for the "Blocked by" row and failed
with a locator timeout.
**Expected behavior**
Each case accepts every valid product behavior and rejects the failures
that matter. A failure names its cause.
**Steps to reproduce**
1. Regrade the retained observation of run 43101125481 for
`cases.create-case` with the old grader: `real-first-morgan-kickoff` and
`required-documents-persist-before-investigation` fail although the
records are correct.
2. Call `corrected({ browser: "Mobile Safari only (bug reproduces only
on mobile Safari)" })` with the old grader: it returns false.
3. Run `task-coordination.block-on-design` when the CEO saves no
blocker: it fails on a UI locator instead of on the missing blocker.
## What Changed
- `cases.update-case-facts` (oracle v8): grade the saved correction
structurally, with no word lists. The browser fact differs from the
case's own setup value, and no field keeps the setup's stale literal
"not yet confirmed", in the saved case and in the reviewer's handoff
context. A person reads whether the wording says mobile Safari.
- `cases.create-case` (oracle case-create.v2): bind the kickoff and its
documents to the CEO chat run that was active when the kickoff was
saved, not to `sourceRunId`. Admit one CEO run on the primary task woken
by Morgan's own case handoff back to the CEO, and add a check that binds
it to that handoff. Run bound is now 2 to 3.
- Shared run policy (`chat-catalog-run-policy.ts`,
`chat-catalog-contract.ts`): new optional `maximumLeadCaseHandbackRuns`
for `source-settled-with-active-work`. The default is 0, so other cases
grade exactly as before. No other oracle version changes.
- `task-coordination.block-on-design` (oracle v9): after the first turn,
stop with "The CEO did not save Design as a blocker on Website" when the
edge is missing.
- Descriptors updated for the three cases.
## Verification
- `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts` on
`integrator-case-create`, `integrator-case-facts`, `runtime-dependency`,
`chat-catalog-run-policy`, `chat-catalog-readiness`,
`chat-catalog-failure-retirement`, `chat-catalog-consistency` and
`chat-ordinary-phase-callers` tests: all pass.
- Calibration in both directions:
- case facts: the retained values of runs 43088288923 and 43107042937
and four rewordings pass. The unchanged setup value, a kept stale
literal (in any field), empty facts and a first turn that left the value
unchanged fail.
- create-case: the retained run 43101125481 regrades to pass. Documents
from another run, a kickoff outside the CEO run, a kickoff by Morgan, a
handback with no handoff, a handback woken another way and two handbacks
fail.
- run policy: an undeclared handback, a lead run woken another way and a
second handback fail.
- `npx tsc --noEmit -p tests/runner-e2e`: no errors in changed files.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`:
unchanged.
- End-to-end: chat eval run linked in a comment below.
## Risks
- Low risk. Only eval code changes. The shared run-policy option is
opt-in, and its default keeps the old result.
## Model Used
- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code
execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
e01ba16bfd |
fix(native): continue the interrupted turn after a relaunch that failed before its runner existed (#20944)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native agent runs survive a server restart: recovery verifies the old runner stopped, then resumes the same provider session and its interrupted turn > - Recovery is a chain of steps. When one step fails once, the next attempt must still be allowed to finish the work > - Here the first recovery step was correct, its runner spawn failed once (EAGAIN), and the next attempt refused to continue the interrupted turn. The run died and needed a manual Retry > - This pull request lets the scheduled same-run resume prove that the failed launch left nothing behind, and then continue the turn > - The benefit is that a single failed spawn during recovery no longer turns a recoverable lost turn into a terminal failure ## Linked Issues or Issue Description **What happened?** Restart-chaos wave `w-20261111T190113-s5303` (local, codex, A1, fault K1 with `--fail-recovery-step runnerd-start:1`) on master `683fd1e9a5`. Run timeline: 1. K1 killed the server mid-turn. Recovery recorded `native.local_process_stopped` for the dead runner and chose `resume_dead_runner`. 2. That attempt wrote `native.process_start_requested`, then the spawn failed with `spawn paperclip-runnerd EAGAIN`. The executor persisted a scheduled same-run resume (`recoveryMode: exact_checkpoint_resume`, `providerSessionEstablished: true`, `checkpointExists: true`, `recoverable: true`). 3. The scheduled attempt resumed the session. The runner reported `provider_turn_lost_on_restore` (recoverable). The run ended `native_provider_terminal_failed` with "Automatic recovery stopped", and the task needed Retry. The same seed with K1 and no injected step passes. Root cause: in `executePaperclipNativeSession` (`native-session-executor.ts`), `resumeInterruptedTurn` is `restartRecovery.kind === "resume_dead_runner" || scheduledResumeContinuation`. The second attempt is not a restart claim, so it depends on `scheduledResumeContinuationAllowed` (`scheduled-resume-continuation.ts`). That decision refused with `local_stop_unverified`: `readNativeLocalProcessStop` reads the newest launch event, and the newest one was the failed `process_start_requested` with no process identity and no stop receipt. The checkpoint evidence was correct; the stop evidence was the gap. The refusal was not logged. **Expected behavior** When a launch fails before its runner exists, a scheduled same-run resume proves on this host that no runner for the run exists and that the launch wrote no runner state, records that stop, and continues the interrupted turn. A live runner for the run still blocks it. **Steps to reproduce** `node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env local --adapters codex --workloads A1 --faults K1 --fail-recovery-step runnerd-start:1 --fault-after-seconds 10 --concurrency 1 --tasks 1 --seed 5303` ## What Changed - `scheduled-resume-continuation.ts`: when the decision refuses only for `local_stop_unverified` on a local target, `scheduledResumeContinuationAllowed` calls the existing `recordUnidentifiedLocalRunnerStop` (the same proof explicit continuations use). It requires that the newest event is this host's launch request, that no process carries `--run-id <runId>`, and that no runner state for the run changed after the launch request. On success it records `native.local_process_stopped` (`basis: unidentified_runner_absent`) in the claim transaction and decides again. All other checks (exact checkpoint, recoverable, session established, checkpoint exists, no replacement, admitted attempt) still apply first. - `native-session-executor.ts`: logs the refusal reason when a scheduled `exact_checkpoint_resume` will not continue the interrupted turn. - `native-scheduled-resume-continuation.integration.test.ts`: two cases. A failed relaunch after a verified stop now continues the turn (`resumeInterruptedTurn: true`, attempt 2). A failed relaunch while a process with `--run-id <runId>` is alive still refuses. The fixture now advances `nextEventSeq` after inserting events directly. - `tests/restart-chaos/README.md`: records what the existing `runnerd-start:1` cell found and what now passes. ## Verification - RED on master: "continues the lost turn after a relaunch that failed before its runner existed" fails with `expected [ false ] to deeply equal [ true ]` (the runtime gets `resumeInterruptedTurn: false`, which is the path that produced `provider_turn_lost_on_restore` → `native_provider_terminal_failed`). - GREEN: `scheduled-resume-continuation`, `native-scheduled-resume-continuation.integration`, `native-exact-resume-failure`, `native-restart-recovery`, `native-runner-restart-recovery.integration`, `native-session-executor`, `native-session-resumption`, `heartbeat-native-restart-suspension` pass (1797 tests). `pnpm --filter @paperclipai/server typecheck` passes. - Chaos rerun of seed 5303 with `runnerd-start:1`: result is added in a comment on this PR. ## Risks - The proof is host-local: it only applies on the host that wrote the launch request, the same as explicit continuations. Remote targets and restart claims are unchanged. - Old saved state: a run already stuck in a scheduled resume after a failed spawn now continues on its next attempt instead of failing. That is the intended recovery; the checkpoint and attempt limits still bound it. - Sibling recovery steps (`teardown`, `directory-lock`) are Daytona-only. The saved waves that armed them (`s5311`, `s5312`) never reached the step, so this PR does not change or prove them. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
0bf3236bc8 |
test(restart-chaos): staging class R cells CR1-CR3 and sandbox capture on every Daytona wave (#20909)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart-chaos harness (`tests/restart-chaos/`) reproduces staging failures before users meet them > - Staging fails with `runner_remote_process_identity_unavailable` when the remote Runner or its provider is gone after a deploy, a stall or a re-wake, and nobody has seen the sandbox side > - The harness had no cell for a provider-only death or a plugin RPC stall, and its sandbox capture ran only on K15 waves > - This pull request adds the staging class R cells CR1–CR3, and captures the sandbox side on every Daytona wave > - The benefit is that a wave that reproduces the refusal also keeps the evidence of why the Runner or provider died ## Linked Issues or Issue Description **What is the problem?** The harness cannot answer why a remote runnerd or its provider dies. Its capture ran only around K15 restarts. It recorded no process start times, sockets or provider log files. It had no fault that kills only the provider or stalls a plugin RPC. **What should change?** Add cells for each staging shape, and capture the sandbox before, during and after each one, on every Daytona wave. **Context** Refs #20907 (keeps runnerd's stderr in the sandbox, and reports it on failed recovery). Refs #20893 (W2 re-wake cells). ## What Changed - `class-r.ts` (new) adds three faults, selected with `--faults`: - **CR1:** a K15 redeploy with a held outage. `--cr1-hold-seconds`, 0–600, default 360. - **CR2:** SIGKILL of only the provider process under the run's `paperclip-runnerd`, inside its sandbox. Runnerd stays alive. By default the controller stays up; `--cr2-hold-seconds` adds a K15 outage. - **CR3:** SIGSTOP of the server's Daytona plugin worker. `--cr3-stall-seconds`, 41–600, default 45. Each CR task must pass the `class_r_fault_applied` check. - `daytona-runner-logs.ts` adds `CLASS_R_PROBE`. It reads process start times and `wchan`, sockets, the runtime file listing, and the tails of the Runner's and the provider's state and log files. It also adds `executeInWaveSandboxes`, the fixed-script runner that CR2 uses. - `main.ts` and `supervisor.ts`: - every Daytona wave captures with the class R probe when a task rests and at wave end; - CR cells also capture before the fault, after the stop, before the restart, and every 15 s for 180 s after ready; - `--sandbox-watch-seconds` captures on a clock for the whole wave. - The README has a new "Staging class R cells" section. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 49 files, 600 tests pass, including the new `class-r.test.ts`. It covers option parsing, planning, the kill command's quoting and its refusal of anything but a run id, plugin worker selection, the watch schedule and the applied check. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos` passes. - Daytona waves run from this branch plus #20907. The results are in the follow-up comment. ## Risks - This change is limited to the harness. CR2 is its only in-sandbox mutation: a SIGKILL of the provider, run by a fixed script only in sandboxes labelled with the wave's company and environment. The script never signals runnerd. - More captures add Daytona toolbox calls on every Daytona wave. Each capture has a 45 s deadline, and a failed capture is recorded and never fails the wave. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file edits, test runs, Daytona waves). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |