mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-20 06:02:18 +02:00
69eb3b6eff4ef00453b623f89abc8a34c8172a9f
5248
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 DottaandPaperclip
|
69eb3b6eff |
perf: reduce repository admission filesystem and write overhead (#16425)
Reduce repository preparation latency by resolving each distinct workspace path once per scan, loading overlapping root provenance, and batching source retention writes. Preserve company authorization, fresh alias checks, and preparation timing. Add a reproducible benchmark and regressions for source identity retention, symlink retargeting, and bounded filesystem and database work. Co-Authored-By: Paperclip <noreply@paperclip.ing>canary/v2026.1015.0-canary.14 |
||
|
DottaandPaperclip
|
5a9e6b6315 |
fix(grok): avoid false file recovery after Stop (#16427)
## Thinking Path > - Paperclip lets people manage AI agents and their work. > - Grok Bot uses a managed Runner and a fixed external work session. > - Its admitted contract gives it no access to Paperclip workspace files. > - The Runner still has a sync descriptor to support durable recovery. > - Cancellation treated that descriptor as proof that files needed repair. > - This pull request uses the admitted access contract while preserving ownership settlement. > - Users get the correct recovery action after Stop, with native file safeguards intact. ## Linked Issues or Issue Description Refs #16219 and #16422. The duplicate search found no open Grok workspace correction. **What happened?** A real Grok Stop test cancelled its run, fenced its assignment, denied a stale progress update and cleaned up the sandbox. Paperclip also created a file-repair hold. The recovery panel told the user to recover workspace files, although the admitted external contract has workspace access set to none. **Expected behavior** Keep the stopped session fenced and require inspection before new work. Keep failed-run ownership settlement. Do not invent a file export obligation for a session that cannot access the workspace. Native sessions with writable or unknown workspace access must keep their file safeguards. **Steps to reproduce** 1. Connect a Grok Bot and assign a task. 2. Let it accept the task and save a Paperclip checkpoint document. 3. Use Stop while it is working. 4. Inspect the task recovery panel. It asks for file repair after sandbox cleanup succeeds. **Paperclip version or commit** Observed on Core |
||
|
DottaandPaperclip
|
57fa6b358d |
fix(observability): separate workspace finalization and copyback spans (#16428)
Give native workspace finalization its own span name. Keep copyback and commit as separate child measurements, with an exported-span regression that preserves the workspace operation result. Validation: 40 runtime/tracing tests and server typecheck passed locally. Final-head CI, E2E and security checks passed. Apex review: 5/5, zero findings and no unresolved threads. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
cccde7122b |
Trace execution preparation and durable response persistence (#16419)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The server prepares an execution environment before it starts an agent. > - Operators need to distinguish preparation, provider wait, and response persistence. > - Existing preparation phases start separate traces, and the native session trace ends before the final response is saved. > - This pull request adds an execution-attempt trace and detailed spans at the existing work boundaries. > - The benefit is a measurable startup and settlement path without changing execution policy. ## Linked Issues or Issue Description **What existing behavior does this improve?** Operator-configured OpenTelemetry tracing for agent preparation, startup, and durable response storage. **Subsystem affected** Server orchestration, native Runner startup, and the Honeycomb run link. **Current behavior** Preparation phases start independent roots. Workspace lock waits, authorization groups, runtime asset work, provider-pack fallback, and the gap after provider completion are hard to attribute. **Proposed behavior** Use a server-owned async context for each execution attempt. Parent nested work to that attempt. Link the separately sampled native session because its ingress and queue history can begin earlier. Record transaction entry, lock waits, grouped authorization, asset preparation, artifact fallback, copyback, and response commit. **Reason and benefit** Operators can identify expensive work from recorded boundaries instead of inferring the cause from elapsed time. Nested and concurrent durations remain visible and must not be added together. **Breaking changes** No API or execution-policy change. New Observability span names and closed count attributes are documented. The existing local run-performance schema also records the additional provider-pack and harness measurements. Refs: #16178, #16205, #16330. ## What Changed - Add `task.execution_attempt` with same-run async ownership, independent root sampling, and native-session links. - Measure workspace transaction entry and settlement, advisory and row waits, source scans, reader groups, notifications, and writes. - Measure runtime selection and each concurrent instructions, skills, and MCP branch. Separate manifest, cache, materialization, and verification work. - Measure provider-pack discovery, verified reuse or upload, final verification, and remote harness state checks. - Measure workspace settlement, accounting, liveness, response authorization, comment commit, and publication enqueue. - Preserve the silent endpoint-off path, including when another library has registered a tracer. Bound repeated detail spans to 16 per name per attempt. - Extend the Honeycomb run link and document the actual coverage and remaining gaps. ## Verification - After resolving the conflict with retained-reader batching on `b98cdaea531c8447637cd4dcc19c61b3f5f41844`, all 61 targeted query, authorization, workspace-preparation and trace-boundary tests passed. The server typecheck passed. Reader tests still bound query count, avoid loading retained run payloads and enforce fresh project grants on every publication attempt. - Before that base integration, the tracing/exporter group passed 855 tests with 2 optional-SDK skips across 10 files and no inherited OTLP endpoint. It covers real PostgreSQL waits/rollback, durable response commit, assembled heartbeat preparation, concurrent parentage, sampling and tracer failure isolation. Native workspace/history tests passed 23 cases. These results remain supporting evidence for the unchanged tracing paths. - Full workspace typecheck and build passed during implementation. Full sharded CI, release validation, E2E and Apex 5/5 passed on prior head `8cebad80f3856dbcff6f3a6d5d34880ff77a20f2`. Fresh CI, security and Apex review are required on the current head before merge. - `pnpm check:token-gates` passed. No styles changed. - Earlier CI runs exposed exact-source-expression assertions changed by wrappers, missing run hashes on transaction-boundary spans, and sampling fixtures without an enabled endpoint. The fixes preserve the calls, add the declared hash and span cap, and enable tracing explicitly in the sampling fixtures. Focused repairs passed; the full prior-head CI passed after those repairs. Earlier failed run: https://github.com/paperclipai/paperclip/actions/runs/38729131366. - The full local `pnpm test:run` was stopped after the base advanced during that process. Its mixed-version results are not final-head evidence. A full local-suite pass is not claimed. - The old-head Superagent scan stalled at 35%. The current head has a fresh scan; no security pass is inferred from the old pending result. - No deployed Honeycomb acceptance is claimed by this code change. ## Risks - Enabled tracing creates more spans. Repeated detail spans are bounded, and inventories use group spans. - The attempt and native session remain linked traces. Historical queue time can precede the attempt, and nested intervals overlap. - A committed comment proves durable server storage. Publication enqueue and realtime dispatch do not prove external delivery or browser paint. - New custom attributes use only the existing run hash and validated file/byte counts. No SQL, paths, permissions, content, or exception text is added. ## Model Used OpenAI GPT-6 through Codex. The serving model ID and context window size are not exposed in this session. The agent used reasoning, tools, code editing, and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
7420af26fc |
fix(ci): finish large Honeycomb profile exports (#16423)
## Thinking Path > - Paperclip uses pull request CI to check changes before merge. > - The merged CI profiler records a selected full PR suite in Honeycomb. > - A real 48-job profile exposed an export failure after Honeycomb accepted only part of the trace. > - The same run exposed three fixture subprocesses that looked like incomplete test launches, one direct reporter without a launcher marker, and one overlong test name. > - The exporter needs to handle temporary ingest rejection without sending accepted spans again. > - This pull request fixes those cases and documents the trace display limit. > - The benefit is a complete, checkable CI waterfall with test-file timings. ## Linked Issues or Issue Description **What happened?** The first live `profile-ci` run on #16423 completed all 48 jobs, but the post-run exporter stopped when Honeycomb rejected part of a batch. The initial trace had no root span. Its coverage flag also reported four unfinished invocations and one invalid timing record. **Expected behavior** The exporter should pace large uploads, retry only temporarily rejected events, and publish the root after every child is accepted. Completed reporters should count as complete. Long test names should not erase test timing. **Steps to reproduce** 1. Add `profile-ci` to a CI CODEOWNER PR with the merged profiler. 2. Run the full PR suite, as in [run 38732093469](https://github.com/paperclipai/paperclip/actions/runs/38732093469). 3. Inspect the separate [failed exporter run](https://github.com/paperclipai/paperclip/actions/runs/38733393370) and the trace coverage fields. **Paperclip version or commit** `12612b06eb4332844ac4c8770a1fb6c6b7eccbc9` (PR run head, based on merged #16378). ## What Changed - Pace Honeycomb uploads and retry only transiently rejected events from each batch. - Preserve overlong test timings with bounded names and count complete reporter pairs without a launcher record. - Keep nested Vitest fixture runs from inheriting CI profiling and creating false coverage gaps. - Clarify ordinary CI access, large-trace display limits, and export behavior in the operator guide. ## Verification - [Full PR suite](https://github.com/paperclipai/paperclip/actions/runs/38732093469/attempts/1): 48 of 48 jobs succeeded; 10m 29s wall time. The selected jobs used the AWS public PR fleet. The protected exporter used a separate GitHub-hosted runner. - [Clean Honeycomb file waterfall](https://ui.honeycomb.io/paperclip/environments/test/datasets/paperclip-ci/trace?trace_id=01ed0c541b0442e613a030ffb6dc8d9b&trace_start_ts=1792065955&trace_end_ts=1792066584): 48 jobs, 822 steps, 2,622 file spans, and `ci.profile_complete=true`. It was reconstructed from that run's 45 timing artifacts after identifying three nested fixture records. The full trace also recorded 44,497 tests; Honeycomb's chart omits some deep test spans because it displays at most 32,000 spans per trace. - The original exporter stopped after a partially accepted batch. A local replay of that batch succeeded. I used the authorized local test-environment key to complete the original upload and create the file-level view. No key was added to this branch or PR. - `node --test .github/scripts/tests/ci-profile.test.mjs scripts/ci-profile/reporters.test.mjs`: 18 passed. - `pnpm exec vitest run server/src/__tests__/vitest-chat-shards.test.ts`: 1 passed. - `git diff --check`: passed. - The local `pnpm -r typecheck` and profiled fixture rerun could not start: the package-manager shim could not verify the `pnpm@9.15.4` registry signature because its fetches failed. `pnpm test:run` and `pnpm build` were not run locally; hosted CI is pending. - The updated exporter has not yet run from protected `master`; this PR remains draft until CI and review finish. ## Risks - A 48,000-span export now takes about three extra minutes due to pacing. Retry limits still fail a persistently rejected batch and keep the root unpublished. - Honeycomb may retain duplicates from an already partial export. The clean file-level evidence trace uses a separate trace ID. - The API key stays only in the protected `ci-profiling` environment. PR jobs write timing artifacts but never receive it. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — see `CONTRIBUTING.md`. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, code execution, GitHub API, and browser inspection. The runtime does not expose the exact serving model ID or context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs: #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>canary/v2026.1015.0-canary.13 |
||
|
DottaandPaperclip
|
ea4aa185b0 |
perf: batch retained-reader authorization reads (#16424)
Batch retained-reader project checks through the existing multi-actor read API. Share raw database reads within one call, retain actor-specific privacy and supplied membership data, and avoid loading large saved-run payloads for identity checks. Keep current-reader writes, distinct saved-run policies, company root fences and later-call revocation checks. Add query-count and access-control regressions. All 255 affected cases and full repository typecheck/build pass. Hosted response-time acceptance remains separate. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
7f9d3d1929 |
fix(grok): avoid routine wakes for proactive admission (#16422)
## Thinking Path > - Paperclip lets people manage AI agents and their work. > - Grok Bot can request governed work from its own conversation. > - Managed Runner admission can take longer than the initial MCP response. > - The dispatcher treated a delayed offer as a reason to start another routine. > - The original conversation was already waiting for that same work. > - This pull request keeps admission in its original request and cancels old prepared echoes. > - Users get one task interaction while normal assigned-task wakes remain available. ## Linked Issues or Issue Description Refs #16219 and #16394. The duplicate search found no other proactive Grok correction. **What happened?** A real idle Grok conversation created one Paperclip task and reached accepted completion. During admission, the dispatcher also sent a routine webhook after the offer was ten seconds old. Two acceptance requests and two title writes were recorded. The extra routine can compete with the original conversation. **Expected behavior** Immediate and delayed Bot-initiated admission stay with the original request. The Bot checks the same request ID and arguments until admission succeeds or fails. A new routine must not echo that work back to the Bot. **Steps to reproduce** 1. Ask an idle Grok Bot to create and complete a Paperclip task. 2. Let managed admission return pending. 3. Leave its offered assignment unaccepted for more than ten seconds. 4. Run the dispatcher. It sends another routine webhook for that proactive offer. **Paperclip version or commit** Observed through the actual Grok client on Core |
||
|
DottaandPaperclip
|
6acec4266e |
feat(ci): profile full CODEOWNER PR suites in Honeycomb (#16378)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Its PR workflow runs dozens of parallel build and test jobs. > - Individual job durations do not show the elapsed cost of the whole suite. > - Maintainers need one shared timeline to find slow work and repeated setup. > - This pull request adds full-suite profiles for selected CI CODEOWNER PRs. > - Honeycomb shows each job, step, test file, and supported test on that timeline. > - The benefit is evidence for changes that reduce total CI time. ## Linked Issues or Issue Description **What existing behavior does this improve?** The main PR workflow has about 48 jobs. Maintainers need to compare their overlapping work and find which jobs delay completion. **Current behavior** GitHub shows job and step durations separately. Test files and tests are not joined to a full workflow trace. **Proposed behavior** A CI CODEOWNER adds `profile-ci` to a PR. The tested code stays on that PR branch. All main PR lanes run, including lanes normally skipped for stack position or release notes. A trusted post-run exporter sends one wall-clock trace to Honeycomb. **Reason and benefit** The trace supports decisions about which CI work to speed up or remove. Ordinary PRs do not export traces. The roadmap and related-PR search found no duplicate CI profiling work. ## What Changed - Add label selection and a full-suite override. Verify the PR author, original actor, and rerun actors against the trusted `.github/**` CODEOWNERS rule. - Record Vitest, Playwright, and Node test file/test timings in local artifacts. Keep ordinary reporter behavior and failing exit codes. - Add a trusted `workflow_run` exporter. Use actual GitHub timestamps for all jobs, setup, steps, waiting, and teardown. Mark missing test data and partial reruns. - Keep the ingest key out of PR jobs. Store it only in the `ci-profiling` environment, restricted to `master`. The exporter reads bounded JSON and never executes artifact code. - Record expected test invocations outside reporters so a missing invocation cannot appear complete. Preserve `node --test` on older PR checkouts. Flush large shard inventories before exit. - Add security, timing, routing, and real-reporter checks. Document operation and activation. Protect the exporter with CODEOWNERS. ## Verification - 509 focused workflow/reporter/shard checks passed. They include outside authors, outside rerun actors, bots, stale revisions, malicious ZIP paths, incomplete reports, and partial ingestion failures. - Final revision `3bf5d9017d` adds multi-file qualification and fixes short-file step parenting and duplicate Node file-wrapper events. Greptile reviewed this exact revision at 5/5 with no unresolved threads. [Main PR CI run](https://github.com/paperclipai/paperclip/actions/runs/38728040302): the initial attempt had 54 successful checks, 3 intentional skips, and one serialized-shard failure from an unchanged workspace-authorization test exceeding its 15-second timeout. The single failed-shard retry passed: the final result is 55 successful checks and 3 intentional skips. The same timeout reproduces locally with profiling disabled; both assertions pass with a diagnostic 60-second timeout override (about 23 seconds total). No source or CI timeout was changed. - A dry run against real GitHub attempt 2 of run `38677445871` produced exactly 3 executed jobs, 40 spans, and 510 seconds, excluding 45 carried-over jobs. The partial trace remains explicitly incomplete; this validation did not upload data. - `actionlint` passed for all four changed workflows. `git diff --check` passed. - Full Node 24 `pnpm -r typecheck` and `pnpm build` passed. The local full `pnpm test:run` was stopped after it reported a setup-token timeout and a distribution-plugin-catalog fixture failure. The 30 setup-token tests passed on focused rerun. The plugin fixture fails at `rmSync` of a directory symlink on this macOS/Node 24.11.0 host; the same failure reproduces with the unchanged base Vitest configuration and unchanged test/service files (10 other tests pass). The local full test suite is therefore not claimed green. - A real root-config Vitest probe passed 13 tests with profiling both off and on. Multi-file Vitest, Playwright, and Node probes each verify two distinct filename rows and four correctly parented tests, including repeated test names across files. Separate passing probes exercise the full exporter hierarchy `workflow → job → step → filename → tests`. Intentional failing probes preserve failing exit status. - Verified all 48 measured jobs in latest-head run `38728040302` used AWS: one `paperclip-ci-routing-x64` gate and 47 `paperclip-public-pr-x64` jobs. Profiling preserves AWS authorization and its GitHub-hosted fallback. The separate privileged exporter runs on `ubuntu-latest` outside the measured suite. - [Live Honeycomb baseline](https://ui.honeycomb.io/paperclip/environments/test/datasets/paperclip-ci/trace/oaPMzw8zGwG): 48 jobs, 731 spans, 479 seconds. It matches [GitHub run 38671274294](https://github.com/paperclipai/paperclip/actions/runs/38671274294). This historical trace has job/step data only; it is not proof of the future label-triggered test-reporting journey. - Verified the dedicated Honeycomb key is ingest-only with dataset creation disabled. Verified the GitHub environment allows only the `master` branch. No repository-level secret of that name exists. Both credential values were scanned absent from source and branch history. After merge, add `profile-ci` to a CI CODEOWNER PR, wait for the entire PR run and exporter, then follow the run-summary link. Collapse job children to compare parallel lanes; expand a job to inspect its steps and tests. ## Risks - Activation requires merging the trusted workflow definitions to `master`. This PR continues using the current trusted reusable workflows until then. - Selected runs produce many spans and artifacts. Oversized artifacts fail export instead of truncating data. Missing or cancelled reporters are marked incomplete. - GitHub timings have second precision. Node test times use reporter receipt time. Rust retains job/step timing but lacks per-test wall timestamps. - CI CODEOWNERS are individual users in the explicit `.github/**` rule. Unsupported team rules and identity lookup failures disable profiling. - An interrupted upload can leave partial data. Retrying uses the same span IDs, but the Events API does not promise deduplication. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, code execution, repository tools, and browser verification. The runtime does not expose the exact serving model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
efcc8a7d89 |
test: bound command snapshot forwarding checks (#16421)
Keep the command snapshot option test focused on forwarding and real file copyback. Use inline delegates in that test, and retain the separate real-worker suites and original deadlines. Validation: 124 affected tests, full typecheck/build, exact-head CI/E2E/Superagent and Apex 5/5. Negative controls reject lost command and restore forwarding. Failed earlier merge-readiness evidence remains in the PR. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
250429942e |
fix(agents): report actionable lifecycle failure steps (#16418)
## Thinking Path > - Paperclip helps people manage AI agents for work. > - New agents must pass setup and execution checks before they can run. > - The lifecycle worker saved the same retry message for every failure. > - Users could not identify the failed step or correct its configuration. > - The UI also described failed work as work still in progress. > - This pull request names the failed step and gives a safe corrective action. > - Users can identify configuration problems and understand automatic retries. ## Linked Issues or Issue Description **What happened?** A new hire showed a paused status and “The lifecycle step failed. Retry the operation.” Repeated retries did not explain the failure. A successful retry cleared the original error. This change cannot recover an error that an older server already cleared. **Expected behavior** Name the failed setup step. Give a useful action for known execution test failures. Explain automatic retry. Keep provider output and credentials out of lifecycle errors. **Steps to reproduce** 1. Create a disposable agent with an unavailable model or invalid execution command. 2. Open its profile while verification fails. 3. Inspect the error, correct the configuration, and select Retry now. 4. Confirm that the failure disappears after the agent becomes ready. Related change: https://github.com/paperclipai/paperclip/pull/16155 shows pending setup progress in creation and assignment. This change improves the cause and action shown when setup fails. ## What Changed - Save the failed lifecycle phase and step in the existing lifecycle error field. - Classify authentication, model, command, directory, timeout, and response failures from execution test check codes. - Use fixed error messages. Do not retain raw provider messages, check details, or credentials. - Share the allowed failure codes with the lifecycle log. - Show failed-state labels and automatic retry guidance. Rename Retry to Retry now. - Extend worker, database, and UI regression coverage. Update the agent verification maps. ## Verification - Passed 105 focused tests across lifecycle diagnostics, database lifecycle commands, saved harness verification, and AgentLifecycleStatus. - Passed server TypeScript compilation with `pnpm --filter @paperclipai/server exec tsc --noEmit`. - Passed UI typecheck and production build. - Passed module boundary checks, token gates, character sync, map validation, and `git diff --check`. - Used Playwright to inspect the production component with a simulated model failure. Confirmed the failure text, retry guidance, retry callback, and recovered state. Captured a screenshot. This component check does not prove a live provider run. - Attempted `pnpm -r typecheck` and `pnpm build`. Both require Rust cargo, which is not installed in the local execution environment. - Started `pnpm test:run` locally. No final local suite result was recorded before handoff. The complete PR CI workflow passed. - Passed all latest-commit CI gates, including typecheck, build, general and serialized tests, Rust checks, and all eight browser test shards: https://github.com/paperclipai/paperclip/actions/runs/38726198774 - Greptile scored the latest commit 5/5 with no findings. There are no review threads to resolve. ## Risks - Failure classification depends on existing adapter check codes. Unknown codes use a safe fallback message. - A known code can describe only the checks reported by the adapter. - No database migration is required. The error field remains a string. - Retry schedules, leases, readiness, approvals, and budget controls keep their current behavior. - Full build verification requires a runner with Rust cargo. ## Model Used - OpenAI GPT-6 through Codex. Used reasoning, code editing, browser checks, and tool execution. The runtime does not expose the exact deployment ID or context window. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
7fc2dc8b32 |
fix: isolate workspace snapshots from controller work (#16414)
Move native sandbox snapshot captures off the controller callback queue while preserving the existing capture and restore contract. Keep legacy adapters inline, bound captures to two workers, and verify cleanup and reordered worker completion. Validation: 461 affected cases, full typecheck/build, exact-head CI/E2E/Superagent and Apex 5/5. The full local canonical suite remains running; no full local pass or under-30-second staging response is claimed. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
Devin Foley
|
8365872743 |
fix(codex-local): retry ACP model capacity failures as transient upstream (#16407)
Classify typed Codex ACP `service` failures about model capacity or server overload as `codex_transient_upstream` / `transient_upstream`, so the existing bounded transient retry and continuation recovery apply. In `unexpected` Sentry mode, skip only runs carrying that full classified evidence; `all` mode still reports them. |
||
|
DottaandPaperclip
|
13e5128164 |
docs: preserve runtime and verification lessons (#16379)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Its control plane shares execution capacity across requests and agent orchestration. > - A dependency can return a small value while doing slow or repeated work underneath. > - Tests can check that value while mocks hide the behavior that affects other work. > - A later passing retry can also leave an earlier material failure unexplained. > - This pull request records general rules for dependency work, verification, and acceptance. > - The benefit is a durable review habit that applies beyond the runner incident. ## Linked Issues or Issue Description Refs #15683. Refs #16335. The staging runner incident exposed a gap in how we establish the operational cost of a dependency call. The original platform check read a small field from a process report. The call could block shared execution, and value-focused mocks did not exercise that behavior. The fix replaced that probe with bounded asynchronous work. This proposal addresses the reasoning and evidence gap, without assigning blame or prescribing one API or timeout for future work. The acceptance guidance also preserves material failures after a retry passes. It does not assert that every earlier test failure had the same cause as this incident. Related proposal checked: #14275 covers release sign-off templates. This PR covers implementation assumptions and verification evidence. ## What Changed - Add a scoped rule for dependency work in server, adapter, and runner execution paths. Cover blocking, repetition, cost growth, cancellation, and the effect on unrelated work. - Require evidence that can disprove the consequential assumption introduced by a change. Identify behavior removed by mocks and exercise the relevant boundary. - Register the rule in the Greptile manifest with matching path scope. Existing `AGENTS.md` instructions route coding agents to matching rules. - Add failure preservation to the verification skill. Link it from the repository-wide PR review rule. Keep recovery and resolution distinct. - Include examples and exceptions so cosmetic edits and ordinary pure transformations do not acquire unnecessary test requirements. ## Verification - Passed `git diff --check` and the repository PR template and duplicate-search validators. - Passed a local Python structural check: manifest JSON parses; rule globs and Greptile scopes match; seven representative runtime paths match; five unrelated paths do not; the evidence link and heading resolve; existing agent and reviewer loading instructions are present. - Manually applied the rule to a network-filesystem configuration getter, a repeated remote permission check, and an already-loaded map lookup. The first two need evidence about their new dependency risk. The last does not require fault testing by default. - Attempted `pnpm -r typecheck`, `pnpm test:run`, and `pnpm build`. The installed pnpm launcher could not fetch registry data to verify the requested pnpm version. Using the cached project version reached typecheck, which failed because workspace dependencies are absent (`@types/node` missing). Tests and build remain unverified locally. - Greptile reviewed commit `e56d1ec70f3ebffe697b35cc637975b8d8198681` at 5/5 with no unresolved review threads. CI is still running. These structural checks establish instruction wiring, not future compliance by an agent or reviewer. ## Risks - The main risk is adding ceremony to low-risk edits. The rule limits requests to concrete risks introduced by changed behavior and explicitly exempts unrelated work. - Instructions depend on contributors and reviewers reading and applying them. This change adds no automated runtime regression gate. - No application behavior, schema, dependency, lockfile, or workflow changes. ## Model Used OpenAI Codex, GPT-6 family. The exact serving model ID, context window, and configured reasoning effort are not exposed in this session. Used reasoning, repository inspection, shell tools, and patch editing. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details available in this session) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable (guidance-only change; structural validation described above) - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
72ae600742 |
test(server): reuse comment wakeup route modules (#16410)
## Thinking Path > - Paperclip manages AI agents and their work. > - Task routes decide which agent wakes after a comment or assignment change. > - The comment-wakeup suite reloads the large route module before each test. > - The worker exhausts its memory before all 34 cases finish. > - This PR loads the module once and resets mock behavior between cases. > - The suite keeps every assertion and runs within the existing memory limit. ## Linked Issues or Issue Description Refs #16017. Related fixes: #16396 and #16404 cover other route suites. The [failed serialized shard](https://github.com/paperclipai/paperclip/actions/runs/38706528509/job/116333628474) completes 26 of 34 cases before a 4 GiB heap OOM. It also logs `runner.select is not a function` when real external-object sync reaches the empty transaction fixture. A duplicate search found no open PR for this suite's memory failure. ## What Changed - Move module reset, mock registration, and initial route import into `beforeAll`. - Mock external-object sync at its direct service import. - Reset mock implementations between cases, then restore the fixture defaults. - Keep all 34 test cases and every assertion unchanged. ## Verification - Node 24.21.0, matching the failed CI job. - Unchanged suite with `NODE_OPTIONS=--max-old-space-size=4096`: worker receives SIGKILL after 31/34 cases in this 4 GiB container. - Fixed suite under the same heap limit: 34/34 pass in 5.06 seconds; peak child RSS is 654 MiB. - Fixed suite with a 1 GiB heap and shuffled order (seed 394): 34/34 pass. - Compared all test bodies and assertions against master; they are byte-for-byte unchanged. - `git diff --check` passes. - Workspace typecheck reaches the Rust stage, then stops because this container has no `cargo`. - Master has a stale lockfile. Local setup used `pnpm install --no-lockfile`; Vitest resolved to 5.0.4 and Supertest to 7.3.1. - Local build stops because Node.js headers are missing for the SO_PEERCRED addon. - The local full-suite command was bounded to three minutes and did not finish. - [Greptile gives this exact head 5/5](https://github.com/paperclipai/paperclip/pull/16410#issuecomment-6188812731), with no findings or review threads. - [CI run 38712230280, attempt 2](https://github.com/paperclipai/paperclip/actions/runs/38712230280/attempts/2) passed on this head. All nine serialized shards, general tests, typecheck, build, Rust checks, and e2e checks passed. - The first attempt hit an unrelated legacy-recovery timestamp flake. Its [single retry passed](https://github.com/paperclipai/paperclip/actions/runs/38712230280/job/116349967086). - Current check inventory: 53 successful check runs, two skipped Storybook checks, and a successful Snyk commit status. Reproduce the focused check with: ```sh NODE_OPTIONS=--max-old-space-size=4096 pnpm exec vitest run --project @paperclipai/server server/src/__tests__/issue-update-comment-wakeup-routes.test.ts --pool=forks --isolate ``` ## Risks Low risk. Only test setup changes. Production routes, workflows, and the lockfile are unchanged. The shuffled run checks that mock behavior does not leak between tests. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools, and code execution. This runtime does not expose a more specific model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandClaude Opus 5.5
|
11d69d04c7 |
fix(native-runtime): report remote recovery labels thrown before coordinator save (#16406)
Restart reattachment can fail in the executor before the native coordinator saves its result. The run failure report now reads the closed remote recovery stage and reason from the caught error when the saved result lacks them. The saved result keeps priority and the same closed-vocabulary validation applies. Add a regression for thrown, wrapped-transport, saved-priority, and out-of-vocabulary cases. Full current-head CI and Greptile 5/5 passed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>canary/v2026.1015.0-canary.11 |
||
|
Devin FoleyandPaperclip
|
307c27e3d8 |
fix(native): reclaim task execution before same-run recovery
Restore the original task execution binding atomically during native same-run recovery. Preserve current owners, stops, pauses, assignments, and dependency waits. Isolate candidate failures, bound deferral writes, prioritize undated retries, and allow recovery ahead of queued waiters without premature queue promotion. Add PostgreSQL attachment, ownership, contention, fairness, and queue regressions. Verification: 99 focused tests, workspace typecheck, build, full current-head CI, and Apex 5/5. Co-Authored-By: Paperclip <noreply@paperclip.ing>canary/v2026.1015.0-canary.10 |
||
|
Devin FoleyandPaperclip
|
cd77c1c4dc |
test(server): reuse remaining serialized route modules
Load route modules once per suite while keeping every test case and assertion. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
|
fd057f6ac4 |
fix(runner): keep OpenCode turns alive through oversized SSE events
Retain bounded OpenCode SSE events without restarting the event stream. Preserve canonical tool identity, status, output tails, original UTF-8 sizes, and truthful truncation markers. Bound id-less event deduplication with hashes. Cover fragmented and complete large frames, exact-limit split separators, bulk input retention pressure, truncation-record exhaustion, and unpaired surrogate byte counts. Full current-head CI and Apex 5/5 passed. The unrelated local post-tool answer assertion was reproduced on unchanged baseline source. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-Authored-By: Paperclip <noreply@paperclip.ing>nightly/v2026.1015.0-nightly.0 canary/v2026.1015.0-canary.9 |
||
|
DottaandPaperclip
|
af4191f45e |
fix: keep workspace notices from suppressing final replies (#16400)
Keep stored system notices visible without letting them occupy the final reply slot. Read presentation metadata in live and recovery selection, and cover explicit replies, progress receipts, company boundaries, and native replay. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
|
8d9d2f9ddf |
fix(recovery): preserve deliberate stops and dependency waits
Preserve deliberate assignee blocks and operator stops during dependency and run-release recovery. Attribute native waits to their bound owner. Ignore unrelated edits and shared-workspace exports from another issue. Continue after genuine dependency changes while preserving independently authorized user input. Keep the original contributor's implementation and review history. Add native ownership, unrelated-edit, later readiness, and shared-workspace regressions. The shared-workspace loop was reproduced before the fix. Co-Authored-By: Devin Foley <devin@paperclip.ing> Co-Authored-By: Paperclip <noreply@paperclip.ing>canary/v2026.1015.0-canary.8 |
||
|
DottaandPaperclip
|
c68a483cd9 |
fix: replay prepared warm seeds when generation caching is unavailable (#16399)
Replay this preparation’s verified private workspace seed after failed warm validation, including when the generation cache refuses publication. Preserve host-current semantics and rebuild if the private seed is missing or changed. Verified 89 focused cases, full typecheck/build, exact-head CI/E2E/security and Apex 5/5. Live PAP latency acceptance remains separate. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
5f463dd625 |
test(server): reuse route modules across test cases (#16396)
## Thinking Path > - Paperclip uses server routes to manage agents and their work. > - Route tests check permissions, login sessions, skills, and file downloads. > - Four suites reload large module graphs before every test. > - Their workers exhaust the heap and block dependency updates. > - This pull request loads each route graph once per suite. > - Each test still gets a fresh app and resets its fixture state. ## Linked Issues or Issue Description Refs #16017. Related: #12987 addresses repeated logger transports through a different production change. **What happened?** CI run 38293835699, attempt 2, aborted four server test workers. The affected suites cover attachments, device login, agent skills, and setup tokens. Three failed log tails show heap exhaustion. The separate `db.transaction` warning came from the document-annotation suite, which passed. **Expected behavior** All route cases complete within the worker heap limit and keep independent test state. **Steps to reproduce** Run the unchanged agent-skills suite from master with `NODE_OPTIONS=--max-old-space-size=1536` and `--logHeapUsage`. The worker aborted after 66 of 90 cases, at a reported heap of 1,532 MB. **Paperclip version** The fix is based on master commit `444d5854b0f53b19d138f060a267cfd10171c7b3`. The original dependency PR head is `7b74c3f248731943154547385b1de5eb6562bf01`. **Deployment mode** Local tests and CI. ## What Changed - Move module resets and mock registration from `beforeEach` to `beforeAll` in four route suites. - Load the agent-skills route modules during suite setup. - Keep each test's app, fixture resets, and login cleanup. - Remove two comments that describe the old reset behavior. ## Verification All 193 tests in the four affected suites pass with a 1.5 GiB heap limit: ```sh NODE_OPTIONS=--max-old-space-size=1536 pnpm exec vitest run --project @paperclipai/server \ server/src/__tests__/agent-skills-routes.test.ts \ server/src/__tests__/agent-device-login-routes.test.ts \ server/src/__tests__/issue-attachment-routes.test.ts \ server/src/routes/setup-token-route.test.ts --reporter=verbose --logHeapUsage ``` - Local runtime: Node 25.9.0, Vitest 5.0.4, Supertest 7.3.1, Express 5.2.1. - Master has an existing manifest/lockfile mismatch. Local installation used `--no-lockfile`. Express was restored to its locked version in an untracked test dependency directory. The tracked lockfile is unchanged. - The agent-skills suite stays below 400 MB of reported heap after the fix. - `git diff --check` passes. - Repository typecheck reaches the Runner Rust check and stops because this environment has no `cargo`. - The repository build and normal server typecheck also stop at the missing Rust toolchain. A direct server `tsc --noEmit` attempt ends with SIGKILL. - [CI run 38686964537](https://github.com/paperclipai/paperclip/actions/runs/38686964537) passes on `c744774bbada4e35ce3acf5991ff19cf50405f2f`, including all general and serialized server shards, typecheck, build, Runner checks, and e2e. - GitHub reports 53 successful check runs, two skipped Storybook checks, and a successful Snyk status. No checks are pending or failing. - [Greptile review](https://github.com/paperclipai/paperclip/pull/16396#issuecomment-6185194537) gives 5/5 on the same commit, with no actionable issues and zero review threads. - The full repository test suite was verified in CI; the local container has a 4 GiB memory limit. ## Risks Module state now lasts for one suite. Each case still constructs its own router and resets its mocks and fixtures. The affected suites pass together locally, and all applicable CI gates pass on the published commit. The change contains four test files. It changes no production code, workflow, manifest, or lockfile. Merge this separate fix before asking Dependabot to rebase #16017. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools, and code execution. The runtime does not expose a more specific model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
5bc4e878e7 |
fix: reuse verified seeds after warm workspace fallback (#16398)
Preserve the verified company/workspace seed-cache scope when a warm sandbox requires host-current staging. Keep remote identity checks, archive digests and rebuilding for missing or corrupt cache generations. 85 focused cases and full repository typecheck/build pass. Exact-head full CI/E2E, Superagent, legacy Snyk and Apex 5/5 pass with no unresolved findings. No staging response-time claim is made. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
79c34058cd |
fix(native): retain unexported workspaces after execution failure (#16397)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native agents can edit files in a remote sandbox before they submit a result. > - A process identity or startup failure can end the run before file export. > - Cleanup can then delete the sandbox and its unpublished files. > - This pull request retains the exact source and records a file repair action before cleanup. > - Operators can recover the files without replaying the failed turn or changing the task owner. ## Linked Issues or Issue Description **What happened?** A native remote process had completed local edits when process identity recovery failed. The run still had a prepared workspace reference and no accepted result. Cleanup deleted the source sandbox. No file recovery card appeared. A restarted run can hit the same path during setup, before the native executor starts. **Expected behavior** Retain the exact unexported source. Show the existing file recovery card. Require source stop and file recovery evidence. Keep task ownership, status, and external action reconciliation separate from file repair. **Steps to reproduce** 1. Run a native agent in a remote ephemeral workspace and make unpublished edits. 2. Fail process identity recovery before result acceptance or file export. 3. Allow failure handling and lease cleanup to run. 4. Before this change, cleanup can destroy the source without a file repair action. Refs: #16341 for related process identity diagnostics. Refs: #16349 for startup ownership fencing. This change addresses file retention after failure. ## What Changed - Record source retention and a file recovery action in the terminal native failure transaction. - Cover restarted preparation failures before the terminal setup writer and cleanup run. - Require the exact company, run, lease, provider allocation, and workspace binding. Require the exact live recovery claim to settle an owned preparation failure. Refuse stale or competing claims. - Reuse the existing stop-only cleanup and file-only reconciliation path. Leave accepted results and scheduled same-run recovery on their existing paths. - Document the retention rule and add database and runtime boundary regressions. ## Verification - All 1,242 tests passed across the full heartbeat recovery suite, native executor, native source retention, runtime teardown, run completion, and legacy file recovery. - The integrated missing-descriptor regression fails before the inner-handler fix and passes after it. The runtime retention and owned restart regressions also reproduce their prior failures. - The tests cover exact source identity, current and stale claims, atomic rollback, cancellation, idempotence, and file-only repair on a completed task. - `pnpm -r typecheck` passed on the final code. - `pnpm build` passed on the final code. - Full repository CI passed on `3945eb2ba5c228c5d4992da02b765dcf2e634cb8`: 54 successful checks and two skipped Storybook checks. - Apex scored 5/5 on that exact commit. Both earlier findings are fixed and all review threads are resolved. - The unsharded local `pnpm test:run` is still in progress. It is not counted as a local full-suite pass; complete suite coverage is verified in CI. ## Risks - Failed runs can retain a sandbox that needs operator cleanup after file recovery. This uses the existing retention policy and recovery controls. - This does not recover files from allocations that were already deleted. - No schema migration, UI change, or telemetry change is required. ## Model Used - OpenAI GPT-6 through Codex. Tool use, code execution, and reasoning assisted this change. The environment identifies the model as GPT-6; it does not expose a more specific serving snapshot or context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
074c67fea8 |
fix(grok): avoid routine wakes for accepted work (#16394)
## Thinking Path
> - Paperclip lets people manage AI agents and their work.
> - Grok Bot receives assignments through routine webhooks and uses
agent MCP tools.
> - Each accepted assignment belongs to the current external
interaction.
> - A pending tool result or comment could start another routine while
that work was still active.
> - Quiet MCP traffic cannot prove that Grok has stopped working.
> - This pull request keeps receipts and feedback durable for the
current interaction to read.
> - New task routines start only for offers that the Bot has not
accepted.
## Linked Issues or Issue Description
Refs #16219 and #16305. The duplicate search also found #12991, which
covers instruction files for local Grok runs.
**What happened?**
The dispatcher could start another routine for an unread result after a
period without a routine reply. It could also wake accepted work for a
task comment. A queued delivery could still start after the Bot had
claimed the offer.
**Expected behavior**
Accepted work stays in its current interaction. The Bot polls the
original operation receipt and reads queued feedback at a safe boundary.
Acknowledgements of a sent wake remain valid after acceptance.
**Steps to reproduce**
1. Accept a Grok assignment and leave it active.
2. Add an unread comment, or complete a tool operation whose call
returned pending.
3. Run the dispatcher after its quiet period.
4. Observe another webhook call. Also queue an offer wake, accept the
offer before sending, and observe that delivery still starts.
**Paperclip version or commit**
Observed during real-client qualification on Core
|
||
|
DottaandPaperclip
|
854ab294cc |
fix: restore repository admission and Claude startup on Linux ARM64 (#16372)
Batch retained workspace admission reads while preserving individual privacy and trust decisions. Preserve bounded saved-policy projections when integrating current master, and qualify the pinned Claude executable on Linux ARM64.
Verified exact head
|
||
|
DottaandPaperclip
|
444d5854b0 |
test(adapter-utils): consolidate outbound restore span coverage (#16115)
Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
a15798e752 |
test(server): reuse seeded delivery workspace fixture (#16114)
Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
1315aa22c2 |
test(ui): retire duplicate company settings checks (#16113)
Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
86816ff965 |
test(adapters): consolidate Grok stale-auth promotion coverage (#16112)
Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
982ff62949 |
test(shared): consolidate ZIP conversion observations (#16118)
Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
5e62927a8f |
test(adapters): consolidate Codex staging permission guards (#16111)
Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
1897080a6d |
test(scripts): retire duplicate provisioning fallback launch (#16110)
Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
5c65525219 |
test(scripts): consolidate stable shard selection cases (#16109)
Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
a05052f77d |
test: retire duplicate heartbeat disposition recovery executions (#16102)
Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
67e45e869e |
test(ui): retire duplicate CompanyExport render checks (#16108)
Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
662a20281d |
test(cli): retire duplicate ZIP codec owner tests (#16116)
Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
0de90e25bf |
test(adapter-utils): share pre-READY work-bound bridge execution (#16098)
Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
1ff08a53cb |
test(tailscale): retire duplicate substituted-listener withdrawal (#16106)
Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
522619744b |
test(ui): retire duplicate IssueChatThread render cases (#16092)
Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
ae6e701821 |
test(ui): share repeated IssueDetail render proofs (#16091)
Co-Authored-By: Paperclip <noreply@paperclip.ing>canary/v2026.1015.0-canary.7 |
||
|
DottaandPaperclip
|
997ea5fd17 |
test(server): retire duplicate process-loss recovery execution (#16088)
Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
3b6d1452f2 |
test(daytona): retire duplicate pack span execution (#16099)
Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
aa6910939b |
test(runtime): share the projectless local lease acquisition (#16086)
Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
d6e4b36c8c |
test(server): move Git redaction proof to its owner (#16081)
Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
|
76164b53cb |
fix(heartbeat): retry a legacy run that lost its lease before dispatch (#16358)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The heartbeat runner executes legacy agent runs. A controller lease watchdog keeps durable ownership of each legacy run, and it aborts the run when it loses the lease. > - #16084 records a lease loss as a `controller_lease` cancellation when the adapter returns or throws. It does not cover a lease loss during setup, before the adapter starts. > - In setup, the abort reason goes to the setup catch. The catch records an opaque `setup_failed` and sends the raw `LegacyControllerLeaseLostError` to Sentry with no cause label. > - Issue continuation recovery treats `setup_failed` as non-retryable. The task moves to `blocked`, but no provider work started. > - This pull request records a lease loss before dispatch as `controller_lease_lost`, with the same `controller_lease` evidence. Recovery retries it as transient infrastructure, and so does the accepted-interaction (plan approval) continuation retry. > - The benefit is that a host lease loss during setup no longer strands the task, and the next Sentry event shows which renewal failure caused the loss. ## Linked Issues or Issue Description Refs #16084 Refs #16355 Refs #16359 **What happened?** A staging server reported `LegacyControllerLeaseLostError: Legacy controller lease lost` from the heartbeat. The run failed during setup with `setup_failed`. Continuation recovery then said "non-retryable failure on this issue's continuation run (`setup_failed`)" and moved the task to `blocked`. A scanner had to move the task back to `in_progress` by hand. **Expected behavior** A lease loss before adapter dispatch is a host infrastructure stop. The run records the `controller_lease` cause, like the adapter-phase path. The run keeps `providerWorkStarted: false`. Recovery retries the continuation with its bounded transient budget, and it does not block the task. **Steps to reproduce** 1. Start a legacy run that has a controller lease. 2. Before the adapter dispatch, make the lease renewal fail: the update matches no row, the query errors, or the lease deadline passes. 3. `controllerLease.assertOwned()` throws the abort reason in workspace preparation or before dispatch. 4. Before this change: the run is `failed`/`setup_failed`, `resultJson` has no `cancellation`, and continuation recovery blocks the task. After this change: the run is `failed`/`controller_lease_lost` with `cancellation.source: "controller_lease"` and a closed cause, and continuation recovery classifies it as `transient_infra`. **Additional context** #16355 handles a different path: a restart orphan that the next boot reaps as `process_lost`. This change handles a live controller that loses its own lease during setup. This change does not find the root cause of the lease loss. The closed cause label in the next Sentry event gives that fact. ## What Changed - `server/src/services/heartbeat/run-completion.ts`: `failRunSetup` receives the execution abort signal. When the abort reason is a `LegacyControllerLeaseLostError`, it writes error code `controller_lease_lost`, adds the `controller_lease` cancellation evidence, and keeps the existing bootstrap recovery facts. The agent goes back to idle, not `error`, because the stop is a host fault. This applies only before provider dispatch: `failRunSetup` also receives `providerDispatchStarted`, and a setup-path error after a legacy adapter or native dispatch began keeps `setup_failed`, so started work is never retried. - `server/src/services/legacy-controller-lease.ts`: add the `CONTROLLER_LEASE_LOST_FAILURE_CODE` constant. - Merged with #16359 (lease-loss timings): the setup failure keeps its `resultJson.controllerLeaseLoss` timings and adds this PR's `controller_lease` cancellation from the abort signal. #16359's setup test now aborts the signal first, as `assertOwned()` does, and expects `controller_lease_lost`. - `server/src/services/heartbeat.ts`: pass the execution abort signal and `providerDispatchStarted` (`legacyAdapterEntered || nativeDispatchStarted`) to `failRunSetup`. - `server/src/services/recovery/service.ts`: add `controller_lease_lost` to the transient-infrastructure continuation codes (3 bounded attempts with backoff). - `server/src/services/heartbeat/retries.ts`: add `controller_lease_lost` to the accepted-interaction continuation retry allowlist, next to `process_lost`. Without it, a plan-approval continuation that lost its lease before dispatch was escalated and blocked before generic recovery could see it. - `server/src/services/heartbeat/run-lifecycle.ts`: add `controller_lease_lost` to the pre-adapter setup codes, so the comment policy does not queue a missing-comment retry. - `doc/run-log-events.md`: document the new error code. - Tests: setup lease loss when the abort reason is thrown, and when a setup step throws a different error after the abort. A unit test shows a lease loss after dispatch keeps `setup_failed`. A classifier test shows `controller_lease_lost` is transient and `setup_failed` stays non-retryable. Two assembled heartbeat tests (`heartbeat-process-recovery.test.ts`) lose the real lease during workspace preparation: one for a plain issue continuation (one `transient_failure` retry) and one for an accepted plan-approval interaction continuation (one `interaction_continuation_infra_retry` retry). Both check the saved run, no adapter dispatch, and the issue still `in_progress`. ## Verification - `cd server && npx vitest run src/services/heartbeat/run-completion.test.ts`: 25 passed (embedded Postgres), including the new post-dispatch test. - `cd server && npx vitest run src/services/run-cancellation.test.ts src/services/recovery/provider-failure-classification.test.ts`: passed on the first head (29 with run-completion; 25 classifier). - `cd server && npx vitest run src/services/legacy-controller-lease.test.ts`: 13 passed (needs `pnpm --filter @paperclipai/plugin-sdk... build` first). - After merging master (head `2148b27`): `run-completion`, `legacy-controller-lease`, `run-cancellation`, `provider-failure-classification` and `run-failure-report` suites, 310 passed; `heartbeat-process-recovery.test.ts -t "lease|dispatch"`, 31 passed. - Head `c52825c`: `retries`, `run-completion`, `heartbeat-retry-scheduling`, `legacy-controller-lease` and `provider-failure-classification` suites, 134 passed; `heartbeat-process-recovery.test.ts -t "lease|dispatch|interaction"`, 41 passed. - Negative controls: with the lease-loss branch in `run-completion.ts` disabled, the plain-continuation assembled test fails (`setup_failed` instead of `controller_lease_lost`). With the `retries.ts` allowlist change reverted, the accepted-interaction assembled test fails because no retry is scheduled. - Not run locally: full `tsc` and `pnpm build`. The sandbox has a 4 GB memory limit and the OOM killer stops `tsc`. CI is the typecheck and build gate. ## Risks - Low risk. The change applies only when the execution abort reason is a `LegacyControllerLeaseLostError` in the setup catch before dispatch. Other setup failures keep their current codes. - The retry uses the existing continuation recovery budgets and admission checks (agent state, budget, pause). The interaction-continuation retry keeps its existing 2-attempt budget. The retry is a new run. The revoked controller cannot dispatch, because its renewal compare-and-set fails. - A run that the reaper settles first keeps the reaper outcome. `setRunStatusIfRunning` makes the first terminal write win. - Sentry still gets this unexpected stop, with the closed `controllerLeaseLossCause` label. If the lease loss is frequent, the event volume does not change. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass (targeted suites pass; typecheck/build not possible in this sandbox, see Verification) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Devin Foley <139239+devinfoley@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
3ea39f75c1 |
test: consolidate duplicate owner proofs (#16075)
Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
8541dfad29 |
test(e2e): share initial Settings sidebar browser proof (#16083)
Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
9076329149 |
test: consolidate duplicate runtime credential proofs (#16079)
Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
5f160421e3 |
test(server): run status corpus fixtures once (#16078)
Co-Authored-By: Paperclip <noreply@paperclip.ing> |