mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-09 16:02:36 +01:00
d26372dc8a86287783e176067097ffa13d3debcf
5
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 DottaandClaude Opus 5.5
|
7afb57c022 |
test(runner-e2e): repair restart-cell harness drift (#19538)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Runner Product E2E suite proves that agent work survives a control-plane restart > - A restart chaos campaign found three restart cells that fail in the test harness, not in the product > - A harness failure hides the real restart result, so the campaign cannot classify these cells > - This pull request repairs the harness drift in those cells and names the one failure the harness cannot prevent > - The benefit is that these cells now test restart behavior, or fail with a clear cause ## Linked Issues or Issue Description Refs #14811 (connector setup became one screen). Refs #19475 (completion-update restart fixture). **What happened?** 1. `lifecycle-baseline.runner-codex.local.tool-review-restart` clicked a "Save and continue" button. #14811 removed that button. The cell failed during setup. After that fix, the cell also failed after the restart: the expanded review card does not survive the post-restart reload, so "Approve & run" was not visible. 2. `local-session-integrity.runner-acpx-claude.local.structured-question-restart-resume` failed with `page.goto: net::ERR_ABORTED` after the controller restart. The old document's Vite client reloads when the replacement server starts, and this aborts the test navigation. The code tolerated only `TimeoutError`. 3. `completion-updates.runner-acpx-claude.local.handoff-completion-restart` waited for the source agent to run an instructed wait command. Claude replied without the command (2/2 attempts), so the restart never occurred. The cell then failed with a generic timeout. **Expected behavior** Each cell reaches its restart boundary and tests restart behavior. If the boundary cannot be reached, the cell names the cause. **Steps to reproduce** Run `pnpm test:e2e:runner -- --max-parallel 1 --max-automatic-retries 0 --id <cell>` for each cell above on `master`. ## What Changed - `tests/runner-e2e/connection-reviews.ts`: accept the one-screen "Check link" step and the older "Save and continue" step. This uses the same regex approach as `connection-flow.ts`. - `tests/runner-e2e/runner.spec.ts` (tool review): click "Review request" after the restart reload, on the document that acts on the card. - `tests/runner-e2e/chat-restart.ts`: add `gotoAfterServerRestart`. It retries only `net::ERR_ABORTED`, within the caller's deadline and at most 5 attempts. It still accepts `TimeoutError`. It rethrows all other errors. The structured-question restart uses it. The existing sentinel and durable-state assertions still prove the new document. - `tests/runner-e2e/completion-updates.ts` and `completion-update-flow.ts`: add `completionRestartGateSkipped` and `completionRestartGateTimeoutDetail`. The gate poll stops at once with `restart boundary not reached (agent skipped gate)` when no completion turn is active and the turn that carries this task's Done facts (`chatCompletionUpdates`, as in `completionDelivery`) succeeded or published its reply before the gate reports `waiting`. A failed or cancelled turn is not blamed on the agent, because delivery can retry it; the timeout detail names it instead. The post-restart chat navigation now uses `gotoAfterServerRestart`. - A fixture-level gate is not possible without a product hook. The native run finalizer publishes the completion reply inside its own transaction, through the issue service closure. The isolated-server entrypoint can patch only class prototypes and HTTP responses, and the reply uses no tool call or HTTP request. - `tests/runner-e2e/README.md`: document the skipped-gate failure. - Docs: use `pnpm --filter @paperclipai/paperclip-runner run build:runner-binaries` in `tests/runner-e2e/README.md`, `tests/restart-chaos/README.md`, `tests/restart-chaos/main.ts` and `doc/DEVELOPING.md`. - No product code changes. ## Verification - Unit: `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/chat-restart.test.ts tests/runner-e2e/completion-updates.test.ts tests/runner-e2e/connection-reviews.test.ts` (36 passed). - Typecheck: `node scripts/typecheck-test-projects.mjs tests/runner-e2e` passed. - Live cells, local, `--max-parallel 1 --max-automatic-retries 0`, macOS, load average 35–65: - `lifecycle-baseline.runner-codex.local.tool-review-restart`: PASS (first run found the review-card reload drift, fixed in the second commit). - `local-session-integrity.runner-acpx-claude.local.structured-question-restart-resume`: PASS. A first attempt failed before the restart with `PRP command session.open timed out` at load ~36 (environment). - `completion-updates.runner-acpx-claude.local.handoff-completion-restart`: FAIL as designed, twice (before and after the review fix), with `restart boundary not reached (agent skipped gate): completion turn … ended succeeded without the reference gate reporting waiting`. Claude skipped the wait both times, so this cell still does not exercise a restart. ## Risks - Low risk. Test harness and docs only. - The navigation retry could hide a real abort. It is limited to `net::ERR_ABORTED`, 5 attempts and the deadline, and the test still asserts that a new document loaded. - The ACPX Claude completion restart cell still does not exercise a restart when the agent skips the gate. It now reports this cause by name. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edit, browser test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
2b2004bf50 |
test(runner-e2e): gate completion briefs inside the run's own task directory (#19475)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agent Chat returns delegated-task completions to the chat that started them, and the runner E2E `completion-updates` suite checks this with real providers > - The restart cell (`completion-updates.runner-codex.local.handoff-completion-restart`) failed before its server restart with "Timed out waiting for worker waiting while originating chat is idle" > - Since project-optional task workspaces (#15749), a task in a project without a workspace runs in its isolated task directory. A native chat runs in its isolated conversation directory. Native Codex can read only that directory. > - The fixture still wrote its brief wait scripts into the managed project workspace and the agent home. Codex got EPERM on both scripts, wrote the note from the task record, and finished. The worker never reached the gate, so the setup poll timed out. > - This pull request makes the waits name only cwd-relative files and publishes each brief into the run's own isolated task directory > - The benefit is that the restart cell reaches its real boundary again (durable Done, source provider held at the gate, server restart) and tests restart recovery instead of a stale fixture path ## Linked Issues or Issue Description **What happened?** The runner E2E cell `completion-updates.runner-codex.local.handoff-completion-restart` timed out in setup. The run ledger shows both agents report `permission_denied` (EPERM) for `node .../wait-for-brief-*.cjs`. The worker run cwd was `<instance>/isolated-workspaces/<company>/<issue>`. The fixture wrote the script to the managed project workspace and the agent home. **What did you expect to happen?** The worker waits at the brief gate while the chat is idle. Then the cell releases the brief, holds the completion reply at its gate, and restarts the server. **Steps to reproduce** `pnpm test:e2e:runner -- --max-automatic-retries 0 --id completion-updates.runner-codex.local.handoff-completion-restart` on master `6cafcaf89b`. **Classification** Test defect. The failure happens before the restart. Workspace isolation is the intended product behavior. ## What Changed - `tests/runner-e2e/chat-stories.ts`: add `workspaceChatBrief(nonce, timeoutMs)`. It returns a bounded `node -e` command that reads and writes only cwd-relative files, plus `at(dir)` to find the gate and ready files in a workspace. - `tests/runner-e2e/completion-update-flow.ts`: the worker and completion instructions use that command. The fixture reads readiness from, and publishes briefs to, `isolated-workspaces/<company>/<issue>` for each delegated task and the source chat. The setup poll state now records the worker run cwd. Cleanup releases every wait that is still open. - `tests/runner-e2e/chat-stories.test.ts`: regression test. It runs the command from a temporary cwd, checks that the command names no host path, and checks the nonce and timeout bounds. It was red before the helper existed. - `tests/runner-e2e/catalog.ts`: bump the `completion-updates` definition to version 28 and set `workerBriefWorkspace: "run-isolated-task-directory"`. - `tests/runner-e2e/README.md`: describe the new brief location. ## Verification - `npx vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/chat-stories.test.ts tests/runner-e2e/completion-updates.test.ts tests/runner-e2e/catalog.test.ts`: 108 passed. - Live cell, native Codex `gpt-5.6-sol`, local runner: `pnpm test:e2e:runner -- --max-automatic-retries 0 --id completion-updates.runner-codex.local.handoff-completion-restart`. Results are in the Verification comment below. - Two earlier local attempts in this worktree failed in setup with `harness_test_failed`. The worktree did not have a built `paperclip-runnerd`. After `pnpm --filter @paperclipai/paperclip-runner build`, the cell passed. ## Risks - Low risk. The change is in test fixtures only. No product code changes. - The idle, busy, and multiple cells share this flow. They now also use the isolated task directory. That is the same directory the product selects for these tasks. - The agent must run the inline command from its cwd. The instructions say this, and the command fails closed (no gate) if the agent runs it elsewhere. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits). Default context window. Extended reasoning on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
27d35314be |
feat: connect experimental Cursor Cloud to native Runner (#18069)
## Thinking Path > - Paperclip helps people manage AI agents at work. > - Native Runner owns execution and task authority. > - Operators need an explicit native Cursor Cloud choice. > - Tools/files need company, task, session and run scope. > - This connects setup and lifecycle to Runner Core. > - Legacy agents retain their selected runtime/history. ## Linked Issues or Issue Description Depends on #18068. Managed-only API placement and API-owned Stop containment are in #18310. Refs #5664, #11803, #17299, and #17471. No duplicate native integration was found. ## What Changed - Add explicit native setup, Test, saved settings and dispatch with no-prompt account/catalog/callback checks. - Scope MCP to company, agent, task, session and run through authenticated HTTPS. Register verified artifacts/work products; preserve unpriced accounting, completion and budgets. - Connect Question/Plan continuation, cold handoff and admitted fresh sessions; recheck waiting admission under the task lock. - Present audited Stop intent while preserving physical containment proof; bound typed Stop metadata. Preserve title edits through preference/navigation readiness. - Select one live/history representation per run to prevent duplicate settled cards. Integrate upstream task/read-state/retry/schema contracts and repair two test fixtures. ## Verification Core #18068 merged normally as `3910352c923a`. Product head `5a60f541c982` has passing repository CI/build/browser checks, including required `ci / verify` and `ci / e2e`, plus a fresh exact-head Greptile 5/5 review with zero unresolved threads. Two non-required external app checks remain pending: Paperclip Review is queued waiting for its assigned agent; Superagent Security Scan is reviewing the diff. These are not claimed completed. The normal merge endpoint enforces the repository rules; no policy or status override is used. The dependent Top PR is addressing its fresh deferred-Stop review finding. Product `5a60f541c982` passed 270 UI and 20 selected Cloud/native/disk-full tests, UI/server typechecks, tokens and maps. Top’s clean normal merge passed tokens/maps/diff checks; its matrix is pending. Earlier receipts retain Core’s 47 Rust, 4 compiled-native retirement and 19 settlement controls, normal Runner build, and runtime 9af build/recursive typecheck. Reproduced loading/shutdown CI fixture failures passed 7/7 browser and 121/121 unit tests, including a broken-unsubscribe negative control. No timeout/oracle was relaxed. The actual-parent restack adopts 75 mainline paths, including schema/permission and generic runtime changes, beyond the prior two fixture repairs and client projection fix. The full difference from hosted `9af1a5de42a2` is therefore larger than four files; backend source also differs. The merged config-read checks match Core391, and the Cursor placement/runtime/relay/selection modules match frozen9af; selected affected regressions pass. Paid proof keeps its actual 9af source/image identity. The separately tested live/history card fix is not claimed deployed on that host. On 9af, original saved settings survived reopening and fresh no-prompt Test authenticated catalog/model/binding; no new Save is claimed. Fresh Linux admission passed. With `claude-sonnet-4-5`, Artifact downloaded exact 27 bytes and matched 121 canonical identities in complete S3-served history. First Stop’s strict quiet FAILED on a 10.373-second gap despite independent cancellation/no late artifact. One targeted unchanged-sampler retry retained all 24 samples: 35.076 seconds quiet / maximum gap 4.584 seconds, 67 ordered canonical identities, and separate CANCELLED/artifact-absent status after the original 180-second deadline. The pre-reload duplicate settled card is disclosed. Exact-ID cleanup for the three new provider agents is awaiting authorization, so full grades and the fresh Artifact’s post-cleanup download remain blocked. A historical deleted-provider artifact was newly downloaded on 9af; original F226 Question/Plan/reconnect and cleanup proof keep their identities. Retained failures: stale fixture 132/133; E8 48/135 with 87 reset-hook failures; Mac 17 SQL failures/skips; earlier title/startup/custody/Stop/deletion failures with unknown causes/bounded recoveries; `dea023` Stop’s warning/50 duplicate identities; pre-repair ACK and original Core CI failures. WSP’s 251-assertion, 7/8-suite attempt failed physical PostgreSQL teardown; the unchanged wrapper rerun passed 128. Retry alone establishes no cause or full local-suite pass. All paid attempts remain recorded; vendor dollar cost is UNKNOWN. ## Risks Experimental: Cursor independently acknowledges neither the effective model nor instruction digest before the first prompt. Explicit values are submitted and checked through behavior; unpriced provider dollar cost stays UNKNOWN. Callbacks need authenticated reachable HTTPS. Scope, revocation, budgets, completion and cleanup remain authoritative. Operator Stop intent is separate from physical containment. Saved legacy agents retain their selected runtime. Prior shared title-save/deletion failures remain disclosed. ## Model Used OpenAI Codex, GPT-6 family, assisted with reasoning, repository tools, code execution, browser verification, and delegated code review. The serving model’s exact API ID and context window are not exposed. No value is inferred from the model catalog. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used and the limits of available model metadata - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have linked public issues or described the issue in this PR - [x] I have not referenced internal or instance-local Paperclip issues or links - [x] My branch describes the change and contains no internal ticket ID - [x] I have run the scoped local tests reported above; pending gates are stated separately - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation - [x] I have considered and documented the risks - [x] All required CI gates pass on the current head - [x] Fresh Greptile is 5/5 with no unresolved findings on the current head - [x] I will address all reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
83076d7e7c |
feat: return completed handoffs to Agent Chat (#14408)
Return completed Agent Chat handoffs through a durable outbox and scope each generated update to its supplied tasks. Add recovery, browser delivery, result access, and calibrated quality coverage. Validated with two consecutive ten-case Claude/Codex campaigns, all CI checks, and a 5/5 review. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
cea8dda472 |
test: evaluate completion updates after native task handoffs (#13969)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Users can delegate work through onboarding and Agent Chat. > - A completed task does not prove that its result reached the original conversation. > - Existing tests do not isolate completion after the source chat becomes idle. > - This pull request adds four explicit native-runner probes across Claude and Codex. > - The probes preserve the result and reply so we can separate delivery failures from inaccurate answers. ## Linked Issues or Issue Description Refs #13775. Refs #13813. These evals extend native-runner qualification. They measure completion updates before we choose a product change. ## What Changed - Add the opt-in `completion-updates` suite with two stories for each native provider. - Test completion in the existing onboarding task flow and after an Agent Chat handoff becomes idle. - Gate the chat worker on a brief inside its managed project workspace. Prove the source is idle before releasing the worker. - Check durable task completion, saved output, a subsequent source reply, and rendered access to the result. - Preserve replies, task state, screenshots, run events, and a separate semantic review rubric. - Add grader regression tests and update the documented eval contract. - Preserve the suites added on master and include four completion cases in the 306-cell catalog. Production behavior and prompts are unchanged. ## Verification - Passed all 565 eval support tests across 45 files after merging current master: `node node_modules/vitest/vitest.mjs run --config tests/runner-e2e/vitest.config.ts`. - Passed eval TypeScript: `node node_modules/typescript/bin/tsc -p tests/runner-e2e/tsconfig.json`. - Confirmed four selected cells: `node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/launch.ts --list --suite completion-updates`. - Four-cell behavior campaign on source `ad47cf1da2b1e36f19f4227cfeb53998720b0b5b`: https://github.com/paperclipai/paperclip/actions/runs/36072337485. - A screenshot-only follow-up waits for the restored source reply to render after result-link navigation. Its one-cell Claude onboarding verification passed on final head: https://github.com/paperclipai/paperclip/actions/runs/36075716141. Corrected report: https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-36075716141-1/. The original four-cell onboarding screenshots caught navigation loading; its saved reply evidence remains valid. The follow-up again found stale wording: "That work will run next" was posted 38 seconds after the child was Done. The four-cell campaign keeps its original source and measurements. - Published evidence: https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-36072337485-1/. - Suite definition: `afba4d85d6c53d9f64c08b37a2e9cc20481b78f5bd7e2fa012045e2c69444d9d`, version 6. Models: native `gpt-5.6-sol` and `claude-sonnet-5`, local execution, one attempt per cell. All four cleanup checks passed. Onboarding billing coverage is partial; reported zero cost must not be read as a free run. | Story | Automated delivery/access | Separate semantic review | | --- | --- | --- | | Codex onboarding | Pass | Pass: accurate completion reply with an accessible result | | Claude onboarding | Pass | Fail: reply says it will save the note once the task runs, after the note is already saved and the task is Done | | Codex idle chat handoff | Fail | Worker completed and saved the note; no completion reply during the full observation window | | Claude idle chat handoff | Fail | Worker completed and saved the note; no completion reply during the full observation window | Both chat cases positively recorded the source waiting and the worker at the brief gate before release. Both saved outputs include the brief-only start time. The opt-in campaign is red because it exposes current behavior. It is not a required merge gate. The PR does not fix that product behavior. Semantic review is a recorded human/agent assessment of retained evidence; it is not an automated prose-quality judge. - Second campaign: https://github.com/paperclipai/paperclip/actions/runs/36071065098. Codex chat reached the idle boundary and completed its task, then received no completion reply during the full window. Claude onboarding again returned a stale handoff answer. Claude chat exceeded the prior 110-second handoff setup budget; this revision raises that bounded setup window to 180 seconds. - Retained baseline: https://github.com/paperclipai/paperclip/actions/runs/36069427676. Onboarding passed delivery/access for both providers, but Claude gave a stale handoff answer. Chat cases stopped at fixture problems; they do not establish a completion-delivery failure. This revision fixes the workspace path and competing reference requirements. - On the previous head `4023a2a3c28d45c9eb2c42d452ce99ffba5c7b73`, 54 PR checks passed and two were skipped, including typecheck, tests, and build. Broad checks ran in CI, not locally. That head received Greptile 5/5 with no unresolved findings. The unchanged mobile repository-settings browser test passed on one targeted retry after a detached/disabled Save-button timeout. - Merged current master in `9b4491e1f` and resolved the catalog-count conflict. Eval support tests and eval TypeScript pass locally. All individual CI jobs passed on this merge commit, including build, typecheck, server tests, runner checks, and browser shards. The final aggregate check also passed: 54 checks passed and two were skipped. Greptile reviewed this exact commit at 5/5 with no unresolved findings. ## Risks - These explicit probes can expose current product failures. They do not change the default paid test selection. - Mechanical delivery and result access do not establish answer accuracy. The preserved reply still requires semantic review. - A fixture failure before the idle boundary or worker completion cannot establish a completion-update failure. - The handoff setup window lasts three minutes. The worker brief wait is bounded at four minutes. The observation window lasts two minutes after worker completion. It retains later replies without erasing earlier accessible delivery. ## Model Used OpenAI Codex, GPT-6 (`gpt-6-astra`), with reasoning, repository inspection, code execution, and GitHub tool use. The runtime does not expose the context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |