PaperClipAI

0 Fork 0

mirror of https://github.com/paperclipai/paperclip.git synced 2026-11-08 12:33:05 +01:00

Author	SHA1	Message	Date
DottaandClaude Opus 5.5	21640b6d51	feat(connections): gate the built-in Google Workspace connector behind an experimental toggle (#19247 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The Apps catalog lets operators connect external tools, including Google services. > - Paperclip has a built-in Google Workspace connector, but Google has not yet verified Paperclip's Google app. > - The built-in connector also hides the Composio and Arcade Google apps, because native Google definitions claim those app names. > - Users therefore see an unverified connector and cannot find the working aggregator Google apps. > - This pull request puts the built-in connector behind a default-off experimental setting. When the setting is off, the aggregator Google apps appear. > - The benefit is that users get working Google connectors now, and the team can still test the built-in connector on demand. ## Linked Issues or Issue Description Problem Google has not verified Paperclip's Google app yet. The Apps page still offers the built-in Google Workspace connector. The built-in Google definitions also suppress the Composio and Arcade Google apps (Gmail, Google Calendar, Google Drive, Docs, Sheets, Slides, Chat), so users cannot find them. Proposed solution Add the `enableGoogleWorkspaceConnector` experimental setting. The default is off. When it is off, hide and block the built-in connector, and show the aggregator Google apps. When it is on, keep the current behavior. Alternatives considered Delete the built-in connector until verification is complete. We rejected this option because the team must continue to test the connector. Additional context Composio documents Gmail (63 tools), Google Calendar (50 tools), and Google Drive toolkits that use Composio-managed OAuth. These toolkits do not need Paperclip's Google app verification. ## What Changed - Added the default-off `enableGoogleWorkspaceConnector` experimental setting to the shared types, validator, feature catalog, server defaults, and the managed-key list for Paperclip Cloud staging. - Added `experimentalFeatureKey: "enableGoogleWorkspaceConnector"` to the `google-workspace` app definition. The existing generic connector gate then hides the connector in the gallery and blocks its setup, discovery, execution, and agent suggestions when the setting is off. - Added a "Google Workspace connector (Paperclip)" toggle card to Settings → Experimental. - Apps page (`Browse.tsx`): when the setting is off, built-in Google rows without saved accounts no longer claim aggregator app identities. The Composio and Arcade Google apps then show as aggregator rows. The legacy Google slug filter now applies only to native rows, so the Composio `gmail` row is not dropped. - Saved per-service Google connections (Gmail, Drive, and others) are not gated. They stay visible and usable. - Updated `doc/connections/GOOGLE-WORKSPACE.md`. - Tests: added tests for the hidden and visible Apps page states, the settings toggle, and the definition gate. Existing built-in Google Workspace tests now enable the setting. ## Verification - `cd packages/shared && npx vitest run` passes (1055 tests). - `cd server && npx vitest run src/__tests__/connection-management.integration.test.ts src/__tests__/tool-access-service.test.ts src/__tests__/tool-gateway-service.test.ts src/services/gmail-tool-governance.test.ts src/services/google-chat-tool-policy.test.ts src/__tests__/instance-settings-service.test.ts src/__tests__/health.test.ts src/__tests__/instance-settings-managed-overlay.test.ts` passes (864 tests). - `cd ui && npx vitest run src/pages/apps src/pages/InstanceExperimentalSettings.test.tsx src/features/connections` passes. - `tsc --noEmit` passes for `packages/shared`, `server`, and `ui`. The `ui` run shows errors only in the untouched `storybook/support/feature-map` files. - `pnpm check:token-gates` passes. - Manual check: open Apps with the setting off. Search for "Gmail". The Composio Gmail row appears, and the built-in Google Workspace card does not. Turn on the setting in Settings → Experimental. The built-in card appears again. ## Risks - Existing built-in `google-workspace` connections stop running while the setting is off, because the generic gate blocks execution. This is intentional, so that the unverified Google app is not used. Turn on the setting on instances that must keep these connections. - Saved per-service Google connections are not affected. - When the setting is on, the behavior does not change. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, running in Claude Code with tool use (shell, file edits, web fetch). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 14:37:24 +00:00
DottaandClaude Opus 5.5	0c9962a9bc	fix(plugins): replace a retired worker with a fresh process (#19245 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Sandbox providers such as Daytona run as plugin workers that the host manages in `server/src/services/plugin-worker-manager.ts` > - The host retires a worker when it cannot prove that a login pseudo-terminal or a duplex channel route is gone (fail closed) > - Retirement used `killProcess()`, which marks a deliberate stop, so the handle stayed `stopped` and no restart followed > - Every later call then failed with `Cannot call "<method>" — worker for "<id>" is stopped` until the server process restarted > - This pull request makes a retirement replace the worker with a fresh process through the existing backoff timer, with a bound on repeated retirements > - The benefit is that one unconfirmed close no longer makes a sandbox provider unavailable for the whole server ## Linked Issues or Issue Description Refs #19150 What happened? A Daytona plugin worker was retired after an unacknowledged duplex close. After that, every run that used the Daytona environment failed setup with `Cannot call "environmentExecute" — worker for <plugin> is stopped`. 15 runs failed in about 12 minutes. The plugin stayed unavailable until the server restarted. #19150 removed the specific trigger. It keeps fail-closed retirement for close replies that the host cannot confirm, so the same outage can still occur. Expected behavior The retired process, its routes, and its tombstones are never reused. The plugin comes back on a fresh process after a short, bounded backoff. A deliberate stop (plugin disable, upgrade, shutdown) stays stopped. Steps to reproduce 1. Start a plugin worker and open a duplex channel whose close acknowledgement is wrong or missing. 2. Close the channel. The host retires the worker. 3. Call any worker method. Before this change, the call fails with `is stopped` forever. ## What Changed - Add `retireWorkerProcess()`. It kills the process but does not mark a deliberate stop. The three fail-closed retirement paths use it instead of `killProcess()`. - In the exit handler, a retired exit settles every route as before, then counts the retirement and schedules a fresh process through the existing backoff timer (`crashed` → `backoff` → `running`). It emits the existing `crash` event with `willRestart`. - A deliberate stop wins. If `stop()` runs before the exit, the exit ends `stopped`. If `stop()` runs during backoff, it cancels the restart. - Bound repeated retirements: at most 5 retirements in the 10-minute crash window. A successful restart does not reset this count (it does reset the crash count), so a worker that starts cleanly and is retired again cannot loop forever. After the bound, the worker stays `crashed`. - `scheduleRestart()` / `computeBackoffMs()` take the attempt number, so the crash path and the retirement path each use their own counter. - Add an optional `crashRecovery` start option (`minBackoffMs`, `maxConsecutiveRetirements`) so tests can use short waits. - Update the Daytona worker close test: a stalled close now shows `backoff`, then a fresh `running` process with a new pid. ## Verification - Red before the fix: the three new tests in `server/src/__tests__/plugin-worker-manager-duplex.test.ts` (`plugin worker replacement after a fail-closed retirement`) failed with `expected 'stopped' to be 'running'` / `'backoff'`. - Green after the fix. The tests prove: - the retired route and a sibling route on the same worker both settle with `{ exitCode: null }`; - the fresh process has a new pid, a write to a retired route does not revive it, and a new route on the fresh process echoes data; - `stop()` during backoff stays `stopped` with no process; - with `maxConsecutiveRetirements: 2`, the third retirement leaves the worker `crashed` and `call()` rejects. - The real Daytona plugin worker test (`plugin-worker-manager-daytona-close.test.ts`, `stalled-close`) now shows the replacement with the actual provider bundle. The closed lease still rejects on the fresh process. - `cd server && npx vitest run src/__tests__/plugin src/__tests__/managed-plugin src/services/plugin` → 48 files, 649 tests pass. The new tests passed 5 runs in a row. - `npx tsc --noEmit -p server` shows no errors in the changed files. - Not run: a live staging run. This PR does not deploy. ## Risks - A plugin that keeps causing retirements now restarts up to 5 times in 10 minutes (1s, 2s, 4s, 8s, 16s backoff) instead of staying dead. After that, it stays `crashed`, as a crash loop does now. - A retirement now emits `plugin.worker.crashed`, so host-service cleanup runs for the plugin, the same as for a crash. A restart emits `plugin.worker.restarted`. A retirement also counts in `totalCrashes`, so idle sleep treats the worker as it treats a crashed worker. - A call made during backoff fails with `is backoff` rather than `is stopped`. The outage is bounded to the backoff, not to a server restart. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edit, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 08:37:16 -06:00
DottaandClaude Opus 5.5	5498db1072	fix(interactions): relay a verified reply to a self-requested human-only confirmation (#19243 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents ask users to approve proposals and plans with confirmation cards, and users often answer in the conversation > - `resolve-from-comment` lets the responding agent record that verified reply on the card > - Native agents often create their own proposal or plan confirmation as `human_only`, because the `request_human_input` schema offers only that value > - `resolve-from-comment` then refuses with `interaction_human_only`, and the plan and onboarding gates keep the task in `in_review` or `blocked`, although the user explicitly approved > - This pull request adds one narrow relay for that case and keeps every other human-only gate on its card > - The benefit is that an explicit conversational approval works again, and governed, company-capped and quality-review approvals keep their existing controls ## Linked Issues or Issue Description Refs #19073 What happened? TestOps launch https://paperclip.testops.cloud/launch/2694 (GitHub run https://github.com/paperclipai/paperclip/actions/runs/41915965118) failed three Product regression cases for the same reason: `continuation/revision-preserves-approval`, `first-task/clarify-propose-accept` and `first-task/revise-accept`. In each case the agent created a `request_confirmation` with `resolverPolicy: human_only` (`effectiveResolverPolicySource: requested`). The user replied "I approve…" or "Yes, I accept that proposal". The agent's `resolve-from-comment` call returned 403 `interaction_human_only`. The task stayed `in_review` (plan gate) or became `blocked` (onboarding decision gate). Expected behavior When the agent itself asked for a human-only answer, the verified latest reply of the user it serves is that human answer. The agent can record it. Company-capped, governed-action, issue-review and quality-review gates still require the human to use the card. Steps to reproduce 1. A native agent calls `request_human_input` with `interactionKind: confirmation` and `resolverPolicy: human_only`. 2. The conversation user posts an explicit approval comment. 3. The agent calls `POST /api/issues/{id}/interactions/{interactionId}/resolve-from-comment`. Before this change it gets 403. ## What Changed - `issue-thread-interaction-resolution.ts`: the agent actor gets an optional `relayedHumanUserId`. A human-only decision is allowed as `allow_relayed_human` only when all of these are true: - the stored policy is `human_only`, its source is `requested` and its provenance is `explicit` - the resolving agent created the card (`createdByAgentId` matches) - no governed action applies - no additional restriction (issue review) applies - any named addressee user is the relayed user Company caps, the `governed_action` source and legacy rows with an unknown source are still denied. - `confirmation-comment-resolution.ts`: the service sets `relayedHumanUserId` only after its existing comment checks pass: user-authored, not agent-attributed, posted after the card, the latest user message, the same conversation session. It also requires that the author is the card's addressee, the conversation user or the run's responsible user, that the card is not a quality review, and that the company's current cap for the kind is not `human_only` (a tied cap keeps the source `requested`). The same actor goes into the locked accept/reject re-check. The activity entry adds `resolverPolicy: "human_only"` and `resolution: "relayed_human_decision"`. - `routes/issues.ts`: the route pre-check passes the candidate comment author. The service still re-verifies everything under its locks. - `issue-thread-interactions.ts`: carries the relay field into the service's own audience re-check. - Docs: the first-task skill and `api-reference.md` describe the narrow relay. The guidance still says not to add `human_only` to ordinary proposals. ## Verification - `server/src/__tests__/confirmation-comment-resolution.test.ts` (embedded Postgres): 72 pass. - Green: relay through a conversation user and through a run's responsible user. The card is `accepted`, `resolvedByAgentId` and `resolvedByRunId` are set, `resolvedByUserId` stays null, and the audit records `relayed_human_decision`. - Negative controls, each with the card still pending and no audit: agent-authored comment, earlier comment, wrong user, unbound issue/run, `company_cap` source, `governed_action` source, `qualityReview`, `toolAction`, `connectionAuthorization`, issue-review `human_only` restriction, another addressed user, human-created card, another agent's card, inherited company default, and a real capped card created through `svc.create` with a tied cap. The tied-cap test fails as expected when the cap is removed. - Existing negative controls are unchanged, including `human_only` without a bound user and the "rechecks narrowed permissions" retry. - Red check: with the three service files reverted to `master`, the two new green tests fail and the negative controls pass. - `server/src/services/issue-thread-interaction-resolution.test.ts`: 21 pass, including the pure evaluator relay rules. - `server/src/services/native-runtime/confirmation-comment-api.integration.test.ts`: 11 pass. The real route accepts the relayed reply and refuses another user's comment with 403. - Also pass: `issue-thread-interaction-routes`, `onboarding-first-task-admission`, `paperclip-runner-tool-authority`, `runner-api`, `onboarding-first-task-assets` and `company-skills-service` (2,474 tests in total). - Changed files typecheck cleanly. - Live verification: Product regression single-case runs of the three cases are pending after merge. ## Risks - This touches an approval gate. The relay is limited to cards whose creator chose `human_only`. All existing reply-evidence checks still apply, plus a bound-user requirement. Company caps, governed actions, quality reviews, issue-review restrictions and named addressees still deny the relay. - Attribution does not change: the resolver is still the agent and run. The human's comment is kept as `result.commentId` and in the audit. - Legacy rows with a null policy source are not relayed. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 08:31:31 -06:00
DottaandPaperclip	47027e7ba5	fix(routines): retain authenticated agent responsibility on creation (#19222 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Routines save work for later execution under a responsible person's authority. > - An agent can create a routine through its normal REST or native API tools. > - The create route kept the agent creator but dropped the authenticated responsible user. > - The service then selected a parent task's user or a company default, even when the source run had a different user. > - This pull request passes the authenticated responsible user separately from creator attribution. > - Future automatic runs then use the authenticated responsible user saved in the routine revision. ## Linked Issues or Issue Description What happened? An authenticated native agent created a routine without a parent task. The stored responsible user became a different company owner. An unrelated parent task could also replace the authenticated user. The defect changes the identity saved for future automatic work. Expected behavior Creation must use the server-resolved responsible user from the authenticated run or API key. The agent remains the creator. A person who clicks Run now still supplies that manual run's identity. Steps to reproduce 1. Create an active company member and a different company owner. 2. Initialize a native agent run with the member's identity. 3. Use `call_api` to create a paused, self-assigned routine without a parent. 4. Read the routine and its first revision. Before this fix, both use the company owner. Paperclip version or commit Reproduced on master `56517dda6c5b32bc901b7dbf1605eaaa041c5fba`. Deployment mode Local source integration with real HTTP routes, agent authentication, native tool authority, and embedded PostgreSQL. No provider turn is needed for the reproduction. Related work: [#19128](https://github.com/paperclipai/paperclip/pull/19128) adds board controls for saved responsibility. [#15207](https://github.com/paperclipai/paperclip/pull/15207) repairs synthetic identities from built-in installers. Neither fixes this authenticated create path. ## What Changed - Pass the authenticated agent's responsible user as a separate internal actor field. - Select the board user first, then the agent’s server-resolved responsible user, before the existing parent and company fallback. - Keep creator attribution, initial revision attribution, manual dispatch identity, and existing records intact. - Add real native API, HTTP, and PostgreSQL regressions for identity selection, replay, client spoofing, stale claims, revoked membership, cancellation, cross-company calls, API keys, and fallback behavior. - Document the creation and manual dispatch identity rules in the execution contract. Correct the obsolete responsibility sentence in the agent access map. - Make the existing live-retry route fixture wait for its committed comment’s durable wake before draining. Keep all process and saved-owner assertions. ## Verification - Captured a failing native `call_api` test before changing production code. The persisted responsible user was the unrelated company owner. - The same real route and database boundary passes after the fix. - Initial affected routine and identity checks: 131 passed. The existing pending-AI-card fence also passed. - Follow-up routine checks: 118 passed. The owning scheduled-retry suite passed all 31 tests. - Initial CI serialized shard 7/9 failed with a `23503` wake-to-agent foreign key error during test cleanup. A controlled real-route/PostgreSQL pause reproduced that exact error on unchanged base `56517dda6c`. The drain returned before the route registered its deferred comment wake. The same pause passed after the fixture waited for the actual commentId’s durable wake receipt. The original failure is preserved. - Affected server typecheck after the fixture query change: passed. - `pnpm -r typecheck`: passed. - `pnpm build`: passed. - `pnpm test:run`: still running as supporting coverage. The worktree changed during the run: map cleanup, then the six-line fixture wait. Production and routine-test files stayed unchanged. This is mixed-working-tree coverage, not an immutable full-suite pass for a commit. Exact next-head CI remains required. - `node scripts/check-maps.mjs` and `git diff --check`: passed. - Exact head `a12e3f6c42b8ad5826fd58faa3cd9553ea251e09`: all CI gates passed. Greptile and the native substantive review both gave 5/5 with no open findings. The native reviewer did not run tests or exercise shadow-mode authorization. - Live provider execution remains unverified. This change does not grant access to another person's AI account. ## Risks New agent-created routines now use their authenticated responsible user before a parent or company default. This is the intended behavior change. The service keeps its fallback when no bound identity exists. Missing agent-key identity still fails authentication. Missing personal AI configuration still blocks execution. Existing routines, revisions, tasks, grants, and manual dispatch behavior do not change. The fixture waits for a persisted wake record before draining; its original live-process and saved-owner assertions remain. ## Model Used OpenAI `gpt-6.1-sol` with reasoning, tool use, and code execution. The host did not expose the context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-05 06:15:25 -06:00
DottaandPaperclip	833d068741	fix(goals): enforce company boundaries for parent goals (#19203 ) Validate parent goals within the child's company and retain parent references through each write transaction. Validate raw plugin patches with the shared goal PATCH contract and keep goal identity immutable. Add real PostgreSQL and plugin RPC regressions for denied writes, valid edits and concurrent reference checks. Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-11-05 02:55:59 -06:00
DottaandClaude Opus 5.5	2ce5f03cd8	fix(workspaces): name remote restore failures and leave a board recovery action (#19207 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A remote task restores its saved workspace into a sandbox before the agent starts. > - When that restore fails, the run records only `remote_workspace_operation_failed: operation=restore exit=1`. > - The run then fails as `setup_failed`. No retry starts, and a `blocked` task gets no recovery action. > - An operator cannot tell which step failed, and nobody owns the next step. > - This pull request makes the sandbox scripts name their failed step in a closed grammar, and it gives a restore failure a bounded retry or a named board recovery action. > - The benefit is a visible cause and a clear owner. Saved work stays protected. ## Linked Issues or Issue Description Refs #19152 and #19187. Those changes name copy-back (save) failures. Refs #18852 and #19195. Those changes added the bounded bootstrap retry that this change reuses. What happened? A board reply woke an agent on a `blocked` parent task. The run failed during setup 13 seconds later. The only run event was `remote_workspace_operation_failed: operation=restore exit=1`. The run had no workspace operation row. The task stayed `blocked`, and its recovery actions showed `active: null`. The same opaque family also appeared for workspace preparation (`exit=1` only). Expected behavior The run names which restore step failed, without command output or paths. A failure before provider work uses a bounded retry when a retry can help. Otherwise, and when the retry budget is spent, the board gets a named recovery action. The saved version is never changed or overwritten. Steps to reproduce 1. Save a remote workspace, then start a new run on a different allocation. 2. Put a folder at the published workspace path in the sandbox before restore. 3. Observe that the base code records only `operation=restore exit=1`, then `setup_failed`, with no recovery action for a `blocked` task. Paperclip version or commit Base: `87dfba6818`. Deployment mode Legacy runtime with sandbox (Daytona) remote workspace storage. ## What Changed - The prepare, repository, and restore sandbox scripts track their current stage. On failure they print `remote_workspace_<op>_failed: stage=<stage>` plus closed fields: `reason=` (the script's own snake_case refusal), `archive=` (the archive helper's fixed reason), `code=` (errno), `disk=full`, `git=` (a fixed label chosen from Git output), and the existing `exit`/`signal`/`program` fields. - The server accepts that text only when it matches a strict grammar for the same operation. Other output is dropped, as before. - Provider refusals during the restore upload and restore command are recorded by closed labels, like copy-back. - A failed saved-workspace restore raises `RemoteWorkspaceRestoreError` with the server step (`snapshot_lookup`, `snapshot_download`, `sandbox_restore`). It is retryable only when the owner re-reads the same writer, epoch, and saved version with no unsaved work, and the sandbox did not name a refusal reason. - Setup failure handling classifies a pre-dispatch legacy restore failure as `remote_workspace_restore_failed`. A retryable failure uses the existing bounded bootstrap retry. A non-retryable or exhausted failure records a board-owned recovery action with the redacted cause, when stranded-work recovery does not scan the task status (for example `blocked`). - Stranded-work recovery gets a named notice for this code. Generic continuation recovery does not grant it a second budget, and legacy reconciliation does not invent provider actions for it. - `doc/execution-semantics.md` documents the contract. ## Verification - Red: the new transport tests failed on the base code with `remote_workspace_operation_failed: operation=restore exit=1` (and the same for prepare and repository). The new completion tests failed because the run became `setup_failed` with no retry and no recovery action. - Green: `npx vitest run src/services/remote-workspace-transport.test.ts src/services/heartbeat/run-completion.test.ts src/services/remote-workspaces.integration.test.ts src/__tests__/heartbeat-retry-scheduling.test.ts src/services/legacy-execution-recovery.test.ts src/services/recovery src/services/remote-workspace-save-concurrency.test.ts src/services/native-runtime/native-workspace-finalizer-failure.test.ts src/__tests__/native-workspace-finalization-work.test.ts` passes (563 tests). - The transport tests use real sandbox commands. They prove an occupied folder reports `stage=inspect_root reason=occupied_root` and keeps its files. They prove a wrong manifest reports `stage=extract archive=manifest_mismatch`. They prove a missing branch reports `stage=checkout_branch`, and a changed origin reports `stage=verify_origin`. Forged output with a path is dropped. - The real-PostgreSQL cold-restore test proves an occupied root gives a non-retryable `sandbox_restore` failure and keeps the unknown files. - The completion tests cover retried, exhausted, refused, post-dispatch, and `in_progress` cases. - `tsc --noEmit` for the server reports no errors in the changed files. ## Risks - A retry keeps the same restore path. If the cause is deterministic but unnamed, the run uses its two retries before the board action. Named refusals skip the retries. - The board action is recorded only for statuses that stranded-work recovery does not scan, and only when no other action is active. This prevents a second recovery identity for one failure. - Script stderr grammar is new. Old sandboxes run the script that the server sends, so no runner update is necessary. - No migration, deployment, or credential change. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use, code execution, and browser tools for read-only staging evidence. Context window and reasoning mode are host-managed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 02:40:01 -06:00
TonioandClaude Opus 5.5	9e9a628a57	feat(ui): turn Artifacts V1 on for every instance and hide its toggle (#19164 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Artifacts show the output of agent work on the Artifacts page, in the panel Artifacts tab, and in the task thread > - #17228 added the Artifacts V1 presentation behind an experimental setting that is off by default > - The presentation is now ready for all users, so an opt-in toggle is no longer necessary > - A new default alone does not reach all instances, because each experimental write persists every flag > - This pull request removes the toggle and turns V1 on for every instance > - The benefit is one artifact presentation for all users, with an off switch for hosting operators ## Linked Issues or Issue Description Refs #17228 Refs #17263 Is your feature request related to a problem? Please describe. Artifacts V1 shows only on instances where an admin turned on an experimental setting. Other users see the previous cards. Describe the solution you would like Show Artifacts V1 on all instances. Remove the toggle from Experimental settings. Describe alternatives you have considered A new default without a change to the hook. This does not work, because many instances have a stored false that nobody chose, and no toggle would remain to change it. ## What Changed - Removed the Artifacts V1 card from Instance Settings, Experimental. - Changed `enableArtifactsV1` to default on in the schema, the feature catalog, and the settings service. - Changed `resolveArtifactsV1Enabled` to ignore a stored false. It honours false only when the value comes from managed config. - Kept the flag and the previous cards, so a hosting operator can hold V1 off. - Kept the key in the Cloud staging list, so a managed false does not hold V1 off on staging. - Updated unit tests, the Artifacts V1 plan, and the directions-lab Today reference. - Moved the attachment and upload e2e specs and the runner download helper to the V1 rows. - Added V1 tests for a Markdown review: expansion, deep link, and open, raw and download actions. ## Verification - `cd ui && npx vitest run`: one test file failed before its fix. That file tests the previous cards, and it now mocks the hook to off. I reran it with the other changed UI tests, and they pass. I did not repeat the full UI run after the fix. - `npx playwright test --config tests/e2e/playwright.config.ts tests/e2e/text-attachment-tabs.spec.ts tests/e2e/artifact-uploads.spec.ts tests/e2e/artifact-tab-arrival.spec.ts`: 10 tests pass on V1. - `tests/e2e/cloud-auth.spec.ts` failed once in CI on a task heading. It does not touch artifacts, and 10 local runs pass. - `cd server && npx vitest run src/__tests__/instance-settings`: 183 tests pass. - `cd packages/shared && npx vitest run src/feature-catalog.test.ts src/validators/instance.test.ts`: 35 tests pass. - `pnpm run check:token-gates` and the `ui` and `server` typechecks pass. I did not run the full repository check. - In the artifact directions lab, with no stored settings, the Today right panel renders the V1 rows. - I did not test this on a live instance. After merge, confirm on staging that the Experimental page has no Artifacts V1 card and that all three surfaces show V1. ## Risks - All users see V1 after the next release, not only staging users. To revert, revert this pull request. - Users cannot turn V1 off. Only a hosting operator can, through managed config, and not on Cloud staging. - An instance with a managed `enableArtifactsV1: false` stays on the previous cards. The cloud harness source has no reference to this key, but I cannot read the live fleet configuration. - V1 surfaces render before settings load. An operator-held instance can briefly show V1 before the previous cards. - Admins who use the API can still write the flag. The write has no effect on unmanaged instances. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 08:14:49 +00:00
Rockiez Rockiez Devin Foley Paperclip	ea42a2e417	fix(recovery): coalesce duplicate legacy no-replay holds (#17055 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Recovery records keep uncertain execution outcomes from being replayed. > - A settled record can still block replay while it waits for a verified decision. > - Repeated scans can create more settled records for the same failed execution. > - Resolving only one duplicate leaves the task behind another hold. > - This pull request retains the settled hold and applies one verified decision to its exact duplicates. > - The task can continue through the existing delivery path without creating another delivery owner for each duplicate. ## Linked Issues or Issue Description Refs #14600. This change addresses duplicate settled holds and their same-source resolution. The visibility and other deferred-wake paths remain outside this change. Related: #10011 handles unchanged active incidents. This patch handles settled no-replay records. #15285 covers missing reconciliation input. #17031 covers a handoff to a new task owner. External-adapter continuation is covered separately by #15982. ## What Changed - Retain a resolved legacy no-replay decision when a later scan finds the same company, task, cause, fingerprint, and source run. - After a verified reconciliation, resolve duplicate records that also match the selected action's kind. Keep the selected action as the delivery owner for this group. Record the merged action IDs in the activity log. - Preserve independent workspace and system recovery records, existing reconciliations, and records for other sources. - Add PostgreSQL regression tests for repeated scans, historical duplicates, delivery count, repeated resolution, scope boundaries, simultaneous resolutions, and rejected reconciliation rollback. - Update the execution semantics and recovery maps. This patch has no schema migration or adapter-specific code. ## Verification Current head: `3c84bc89325d40bfdda5d28f24ed2fb67d48756c`. It merges `master` at `043f2514e67c50320433a0013fb228a75a51325c` without rewriting the existing commits. The only conflict was in `markExecutionReconciliation`. The resolution retains the upstream transaction and all three queue notifications. Source, sibling, and audit writes use the same transaction. The final merged tree passed: ```sh pnpm exec vitest run server/src/__tests__/issue-recovery-actions.test.ts server/src/services/legacy-execution-recovery.test.ts server/src/__tests__/heartbeat-process-recovery.test.ts server/src/__tests__/execution-control-work.test.ts --maxWorkers=1 # 549 passed; zero skipped pnpm --filter @paperclipai/server exec tsc --noEmit # exit 0 node scripts/check-maps.mjs # passed: 35 areas, 191 capability IDs git diff origin/master --check # passed ``` Three new cases add real PostgreSQL and HTTP-route evidence. Two resolution requests target different duplicate holds while both wait on database locks. They leave one continuation owner, one coalescing audit record, and one queued successor. A live-provider rejection leaves all records unchanged. A checkout rejection after reconciliation writes rolls back the source, siblings, and audit row. The fixture occupies the dispatch slot, so no external model provider runs. The new contention case was also run with only `execution-recovery-resolution.ts` replaced by its parent revision at `65f59e339a9fb6e0635bcbd8c36791bd610326a6`. It failed with two continuation owners instead of one. The source was restored byte-for-byte, and all three new cases passed again before the merge. The final four-suite run above validates the merge with the upstream notification changes. The original positive regressions also failed on that parent revision: three scan-and-settle cycles left four records instead of one, and resolving one historical hold left its two duplicates unreconciled. The scope matrix covers company, task, kind, cause, fingerprint, source run, replay state, protected recovery records, and prior reconciliation. Fresh-checkout collection initially failed because plugin SDK build outputs were absent. After building them, PostgreSQL startup exposed native library symlinks missing from an installation with scripts disabled. The package's symlink setup resolved that failure. The standard server typecheck attempt started a Rust vendor prerequisite build and was stopped; direct TypeScript checking passed. These failed or incomplete attempts are not counted as passes. Full repository build, recursive typecheck, and CI were successful on the earlier `08e5c0285` revision ([CI run](https://github.com/paperclipai/paperclip/actions/runs/39541949702)). They are historical evidence, not results for this merge. The current local proof is the four suites and checks listed above. Full upstream CI for the merge is pending. Greptile reviewed `3c84bc89325d40bfdda5d28f24ed2fb67d48756c` with 5/5 confidence and no actionable findings; that source review is separate from test execution. ## Risks - A match that is too broad could clear an independent hold. Tests cover company, task, kind, cause, fingerprint, source run, replay state, workspace recovery, system recovery, and prior reconciliation boundaries. - The existing issue-lock transaction covers the selected action and its duplicates. The patch does not create a new lock or an automatic retry path. - The patch does not infer that a network or provider failure means no action occurred. Existing reconciliation validation still checks the source, owner, stopped processes, leases, and observed action outcome. - Previously reconciled records and their delivery state remain unchanged. This patch prevents new duplicate delivery owners when it resolves unreconciled duplicates. It does not migrate every historical delivery record. ## Model Used OpenAI Codex, based on GPT-6, assisted with the implementation, tests, and review. It used repository tools and code execution. The exact runtime model ID, context window size, and reasoning setting are not exposed in this session. An independent AGY review also ran; its provider model ID was not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with the available version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have linked the existing issue with `Refs #14600` - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal task IDs or instance details - [x] I have run the focused tests locally and they pass; full upstream CI is pending for the final merge - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Rockiez <7697772+Rockiez@users.noreply.github.com> Co-authored-by: Devin Foley <devin@paperclip.ing> Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-04 23:54:51 -08:00
DottaandClaude Opus 5.5	037c58bb22	fix(daytona): keep the provider auto-stop strictly after the warm runner idle window (#19199 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A warm remote agent keeps its sandbox running after a turn. The host keeps a warm runner for an idle window (`runnerIdleTimeoutMs`, or the agent's `idleTimeoutMs`) > - The Daytona provider also stops an idle sandbox after `autoStopInterval` minutes. The default is 15 minutes > - On staging, the warm idle window was 900,000 ms (15 minutes). The two windows were equal > - A follow-up turn at the end of the warm window resumed a sandbox that Daytona was stopping. Daytona refused the resume in about 4 seconds, and the run failed at setup > - #19195 added a bounded retry for that refusal. It did not remove the cause > - This pull request makes the two windows non-overlapping by construction. Daytona derives its auto-stop for a warm reusable sandbox from the same warm idle value that the host uses > - The benefit is that a follow-up inside the warm window always finds a running sandbox, and idle sandboxes still stop ## Linked Issues or Issue Description Refs #19195 (bounded retry for a refused resume; this change removes the common cause). What happened? On a staging soak, three follow-up runs failed at setup with a refused sandbox resume. Each resume began 15 minutes 8 to 25 seconds after the previous turn released the sandbox. The warm idle timeout was 900,000 ms. Daytona's default `autoStopInterval` is 15 minutes. Nothing in the code kept these two values apart. Expected behavior A reusable sandbox that keeps a warm runner must not auto-stop inside the warm idle window or at its edge. The provider backstop must fire after the host's own idle handling, with a margin. Idle sandboxes must still stop. Steps to reproduce 1. Configure a Daytona sandbox environment with `reuseLease` and a warm runner with `runnerIdleTimeoutMs: 900000` (or an agent with `lifecycleMode: warm`, `idleTimeoutMs: 900000`). 2. Run a turn. Do not set `autoStopInterval`. 3. Before this change: the sandbox is created with `autoStopInterval: 15`, equal to the warm window. A follow-up near minute 15 races the provider stop. Paperclip version or commit `c656b95b6b` (current `master`). ## What Changed - Daytona plugin (`plugin.ts`): add `effectiveAutoStopIntervalMinutes`. For a reusable lease that keeps a warm runner, the effective auto-stop is at least `ceil(warm idle / 1 min) + 5` minutes. The plugin applies the floor only when the host forwards `warmRunnerIdleTimeoutMs`, or when the environment itself selects `runnerLifecycleMode: warm`. An explicit `0` (disabled), a longer configured value, `inherit` with a per-turn agent, per-turn, and non-reusable sandboxes keep the configured value. - Daytona plugin: use the effective value at sandbox creation. On resume, raise a shorter auto-stop on an existing sandbox. This update is best effort and bounded to 5 seconds; the #19195 retry still covers a refusal. The stale-handle refresh threshold keeps the configured interval, so a cached handle is not trusted for longer before the provider confirms the new interval. The warm window is not written to the saved normalized config. - Host (`runner-environment-lifecycle.ts`): add `resolveWarmRunnerIdleTimeoutMsForRun`. It returns the run's effective warm idle window (environment `runnerIdleTimeoutMs`, else the agent `idleTimeoutMs`) for a warm `paperclip_runner` sandbox run. It shares the lifecycle-mode resolution with `resolveRunnerEnvironmentForRun`. - Host (`environment-run-orchestrator.ts`, `environment-runtime.ts`): pass that value as `warmRunnerIdleTimeoutMs` to the sandbox-provider driver on acquisition and on remote recovery resume. The driver adds it to the provider worker config only. Lease identity, reuse matching, and lease metadata are unchanged. - Daytona manifest: describe the warm floor on `autoStopInterval`. - `doc/execution-semantics.md`: document the contract. ## Verification - Red (before the fix, same tests, with the floor disabled): 5 failures in `plugin.test.ts` "warm idle window versus Daytona auto-stop". The staging case failed with `expected 900000 to be greater than or equal to 1200000`. The resume case failed because `setAutostopInterval` was not called. - Red (host): the new `environment-runtime.test.ts` case first failed because the provider worker config had no `runnerIdleTimeoutMs`. The service wrapper dropped the field. The fix forwards it. - Green: `npx vitest run --config vitest.config.ts` in `packages/plugins/sandbox-providers/daytona`: 563 passed. This includes a setter that never settles (resume completes after the 5-second bound) and `inherit` with no forwarded window (explicit `autoStopInterval: 2` stays 2). - Green: `npx vitest run src/services/runner-environment-lifecycle.test.ts src/__tests__/environment-runtime.test.ts src/__tests__/environment-run-orchestrator.test.ts` in `server/`: 242 passed. This includes recovery resume forwarding the admitted warm window. - The regression tests assert the relationship `autoStop >= warm idle + margin` for several warm windows (125 s, 300 s default, 900 s, 1,800 s) and for an explicit `autoStopInterval: 15`. They are not constant checks. - `node scripts/typecheck-low-memory.mjs` passes. `pnpm typecheck` in the Daytona package passes. ## Risks - A warm reusable Daytona sandbox now stays running up to the warm window plus 5 minutes after its last activity, instead of 15 minutes. With a 15-minute window this is 20 minutes. Cost is bounded and idle sandboxes still stop. - The resume-time `setAutostopInterval` call is best effort. If it fails, the old interval stays and the #19195 retry still applies. - Other providers are unchanged. They receive one extra host-owned config key (`warmRunnerIdleTimeoutMs`) only for warm `paperclip_runner` runs, and they ignore unknown keys. - No lease identity change, so existing retained sandboxes are still reused. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through Claude Code with tool use (shell, file edits, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 01:10:19 -06:00
DottaandClaude Opus 5.5	c656b95b6b	fix(environments): retry a refused sandbox resume with the bounded bootstrap budget (#19195 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A warm remote agent keeps its sandbox after a turn, so a follow-up turn resumes the same sandbox and its files > - The sandbox provider auto-stops an idle sandbox (Daytona: 15 minutes by default). The warm native idle window is also 15 minutes > - A follow-up that reaches the sandbox during that stop gets a provider refusal. The host keeps the sandbox ("the lease was preserved and no replacement was created"), which is correct > - But setup then records a generic `setup_failed`. No retry is scheduled, no cause is recorded, and the reply that woke the agent is effectively dropped > - This pull request gives that refusal its own failure code and routes it through the existing bounded bootstrap retry. The retry resumes the same recorded sandbox > - The benefit is that a follow-up continues a few seconds later on the same saved sandbox, instead of stopping the task with no next action ## Linked Issues or Issue Description Refs #18852 (the bounded bootstrap retry that this change reuses). Related: #19152 and #19187 (copy-back causes), #19178 (Daytona errored-sandbox recovery before stop-only cleanup), #19182 and #19180 (recovery holds). This change does not overlap with them: they cover cleanup, copy-back, and recovery after a run. This change covers a resume refusal before provider work starts. What happened? On a staging soak, two follow-up turns failed at setup after a successful first turn: - A board reply woke the agent. The run failed after 4.9 seconds with `setup_failed`: `Reusable sandbox lease "…" could not be resumed; the lease was preserved and no replacement was created.` The issue stayed in review with no recovery action. The reply was not delivered. - An earlier run on another task failed with the same message. Both resume attempts started 15 minutes 8–25 seconds after the previous turn released the sandbox. The first turns used `keep_running` with a warm idle timeout of 900,000 ms. The Daytona default `autoStopInterval` is also 15 minutes. So the resume reached the sandbox while the provider was stopping it. The provider refused within seconds; this was not a long resume timeout. The provider cause was not recorded, so this timing is the strongest evidence. The exact provider state remains an inference. Expected behavior A refused resume before provider dispatch is a temporary infrastructure condition. The host retries it with a small, bounded budget. Each retry resumes the same recorded sandbox. A refusal does not prove that the sandbox is lost, so the host never destroys it or creates a replacement. Only an explicit provider not-found or a workspace identity mismatch with a verified backup permits replacement (unchanged). The run records why the provider refused, without provider text. Steps to reproduce 1. Run a warm remote turn that keeps its sandbox. 2. Make the provider's `environmentResumeLease` throw a non-transient error (for example, a 400/409 while the sandbox stops). 3. Wake the agent with a reply. 4. Before this change: the run fails with `setup_failed`, no retry is scheduled, and the error has no cause. Paperclip version or commit Seen on staging at `5b99bdae`. The touched code is unchanged on current `master` (`f7a4c5678f`). ## What Changed - Move `ReusableSandboxResumeError` to `server/src/services/reusable-sandbox-resume-error.ts` (re-exported from `environment-runtime.ts`). Give it the code `reusable_sandbox_resume_failed`. - Add the provider refusal to the error message by closed labels only: `(cause=<label> status=<http> code=<errno>)`. The labels come from the existing transport classifier, now exported as `sandboxProviderFailureLabel`. Provider text, paths, and IDs are never copied. - Carry the provider status across the plugin worker RPC. A JSON-RPC error has only a message and a numeric RPC code. The worker host now attaches a validated `paperclip/environment-resume-error/v1` envelope (HTTP status and an allowlisted errno only) to `environmentResumeLease` failures. The host re-validates it and adds it to the recorded cause. - In setup-failure settlement, find this error through the orchestrator wrapper. Record `reusable_sandbox_resume_failed` only when provider dispatch did not start and the run has no native session or runner identity (the same guard as `remote_workspace_prepare_timeout`). - Add the code to `isTransientWorkspacePreparationCode`. The existing scheduler then gives two retries (30 s, 60 s) with its issue gate, Stop, pause, and dependency checks. - Exclude the code from generic continuation recovery and from legacy reconciliation (no unknown provider actions; bootstrap evidence proves none started), like the template. - Add a specific stranded-recovery notice: "Saved sandbox could not be resumed". - Document the contract in `doc/execution-semantics.md`. ## Verification - Red (before the fix, same tests): 6 failures. The `fresh` case recorded `setup_failed` and did not schedule a retry. The error had no cause. Legacy reconciliation held the run. Generic recovery granted another budget. - Green: `npx vitest run src/services/heartbeat/run-completion.test.ts src/services/legacy-execution-recovery.test.ts src/services/recovery/ src/__tests__/heartbeat-retry-scheduling.test.ts src/__tests__/environment-runtime.test.ts src/services/remote-workspace-transport.test.ts src/services/heartbeat/retries.test.ts` (in `server/`): 619 tests pass. Also `src/services/recovery/stranded-notice` and `src/services/remote-workspace*`: 411 pass. - Safety regressions: - `run-completion.test.ts`: only a pre-dispatch legacy failure with no native identity gets the code and a retry. `started`, `retained` (native), and an untyped copy of the message stay `setup_failed` with no retry. The recorded error contains the closed cause and not the provider path. - `heartbeat-retry-scheduling.test.ts` (real PostgreSQL): two concurrent schedulers across a restart create exactly one successor per attempt. The budget ends after two attempts. Stop, pause, and a new dependency cancel the scheduled retry. Generic stranded recovery creates no extra run after exhaustion. - `packages/plugins/sdk/tests/environment-resume-error.test.ts`: through the real worker RPC, the `JsonRpcCallError` has no status, and the envelope gives 409 and the errno. Provider text does not cross. Other methods get no envelope. Malformed envelopes are rejected. - `environment-runtime.test.ts`: a throwing resume still preserves the exact lease. The runtime does not call destroy or acquire. The resume throws a `JsonRpcCallError` whose message has no status. The thrown message names `cause=provider_error status=409` from the envelope, and not the provider text. - `node scripts/typecheck-low-memory.mjs` passes. `git diff --check` passes. - Not verified: a live staging run at this revision. The original tasks were not retried. This PR is not deployed. ## Risks - Low. The change affects only runs that fail with this typed error before provider dispatch. Each retry resumes the same lease and inherits the source wake context, so a reply is delivered once and no second executor starts. - If the provider cannot start the sandbox for longer than about 90 seconds, the two retries end and the task gets the specific recovery notice. The sandbox and its saved files are kept. - The error message now has a parenthetical cause before the semicolon. No code parses this message. - An issue in review does not get the immediate stranded notice (unchanged existing behavior). The retry, not the notice, delivers the reply in that case. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, browser reads of staging diagnostics). Extended reasoning enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 00:20:21 -06:00
Devin FoleyandPaperclip	895cee28b3	fix(daytona): recover an errored sandbox before stop-only cleanup (#19178 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Remote agent runs use sandbox provider plugins. Daytona is one of them. > - After a failed run, Paperclip stops the sandbox and keeps its files ("stop-only cleanup"). A task can continue only when every remote lease of the failed run has a provider stop receipt. > - Daytona rejects Stop for a sandbox in its `error` state. A sandbox that filled its disk went to that state. The plugin called Stop on it, so every cleanup sweep failed with the same generic error. > - The lease stayed in pending cleanup without a stop receipt. Each Retry or comment waited forever with reason `remote_cleanup`. A manual Daytona Recover was the only fix. > - This pull request makes the Daytona stop-only hook recover a recoverable sandbox once, with a bound, and then stop it. When it cannot stop the sandbox, it returns a typed failure. The server escalates a terminal failure at once instead of retrying it like a transient error. > - The benefit is that a full-disk sandbox no longer blocks its task, and a sandbox that cannot recover is visible to operators on the first attempt. ## Linked Issues or Issue Description No public issue exists. Bug description: What happened? A task sandbox filled its 10 GB disk. Daytona then reported the sandbox as `state: error`, `desiredState: started` and `recoverable: true`. Its `errorReason` was a Docker `no space left on device` error. The Daytona `stopLease` hook calls Stop on any sandbox that is not `stopped` or `archived`. Daytona rejects Stop in the `error` state. The pending cleanup sweep ran about every 30 minutes, and it failed on every pass. The lease stayed `pending_cleanup` with `cleanup_status=failed` and no stop receipt. Continuation admission (`server/src/services/explicit-native-continuation.ts`) requires a stop receipt on every remote lease of the failed run. So each Retry and comment was deferred with "Waiting for the previous environment to stop. Your message will start automatically." It never started. A manual Daytona `POST /api/sandbox/{id}/recover` fixed it. The sandboxes reached `started` in about one minute. The next sweep stopped them, recorded the receipt, and the tasks continued. Expected behavior Stop-only cleanup recovers a recoverable errored sandbox and then stops it. When the sandbox cannot be recovered, cleanup returns a clear, typed failure that the server can tell apart from a transient error. Cleanup never deletes the retained sandbox, and it never loops on Recover. Steps to reproduce 1. Run a task in a Daytona sandbox until the sandbox fills its disk and Daytona shows `state: error`, `recoverable: true`. 2. Let the run fail, so the server records a `sandboxStopAndRetain` intent and calls `environmentStopLease`. 3. Before this change, Stop fails with `Sandbox failed to stop with status: error` on every sweep, and the task cannot continue. 4. The new tests in `plugin.test.ts` reproduce this with a mock sandbox whose Stop fails from `error`. Paperclip version or commit Observed on a hosted deployment. Reproduced in unit tests on `master` at `6452e964a36`. Deployment mode Managed Daytona sandbox execution with plugin-backed stop-only cleanup. Related: #18172 (Daytona exports under disk pressure) and #19050 (session reset on every Daytona restart path). This PR does not change the resume path in `ensureSandboxStarted`, which already recovers a recoverable sandbox. ## What Changed - Daytona plugin (`packages/plugins/sandbox-providers/daytona/src/plugin.ts`): after `stopLease` confirms the retention policy, a sandbox in `error` goes through `recoverSandboxForStop`. - Recoverable: the hook calls `sandbox.recover()` once. The SDK waits for `started`. The plugin bounds the wait to 20 s, plus the existing liveness margin for a hung connection. Then the hook stops the sandbox normally and returns the usual `stopped` receipt. - Not recoverable: the hook makes no Recover or Stop call. It throws `PluginEnvironmentStopError("resource_unrecoverable")`. - Recover fails: the hook reads the provider state once. A sandbox back in `error` gives `recovery_failed`, or `resource_unrecoverable` if Daytona now marks it so. A sandbox that is still starting, or a failed read, gives `recovery_incomplete`. - The recovered sandbox fails again before Stop finishes (for example, the disk fills again): a fresh read that shows `error` gives `recovery_failed`. The hook never calls Recover a second time in the same call. - The hook never deletes or archives the sandbox. The non-error stop path is unchanged. - Plugin SDK (`packages/plugins/sdk/src/environment-stop-error.ts`, new): `PluginEnvironmentStopError`, `environmentStopErrorData` and `readEnvironmentStopFailure`. This follows the existing `PluginEnvironmentRunnerIngressError` pattern. The envelope holds only one closed reason. The reader validates the reason and derives `terminal` from it. `resource_unrecoverable` and `recovery_failed` are terminal. `recovery_incomplete` is not. - Worker RPC host (`packages/plugins/sdk/src/worker-rpc-host.ts`): forwards that envelope as JSON-RPC error `data` for `environmentStopLease` only. - Pending cleanup sweep (`server/src/services/heartbeat/recovery.ts`): the catch reads the validated envelope. - On a terminal failure, it escalates on the first attempt. It writes the one-time "needs operator attention" warning and uses the existing 30 minute retry interval. Before, this happened only at the fifth attempt. - The failure log uses `errorKind: "provider_stop_terminal"` and includes the closed `providerStopFailure` reason. The sweep still never logs exception fields. - The lease stays `pending_cleanup`. Automatic cleanup continues, and an explicit Retry still bypasses the cooldown. Untyped and transient failures keep their current behavior. - Docs: the SDK README ("Stop failures"), the Daytona README, `doc/plugins/SANDBOX_PROVIDER_CAPABILITIES.md` and `doc/native-workspace-finalization-recovery.md` describe the new behavior. Shared contract note (module boundaries): the SDK envelope is the smallest addition that lets the host tell a terminal provider stop failure from a transient one. The worker host drops all other error data across the RPC boundary, so the existing contracts cannot carry the signal. The only consumer is the pending cleanup sweep. When a provider does not throw the new error, or an older host ignores it, behavior is unchanged. This PR does not touch `server/src/services/workspace-recovery.ts` or `sandbox-stop-and-retain.ts`. ## Verification All commands ran in a clean worktree from `master`. Vitest ran from the real path. - Red on `master` (only the new tests added): - Daytona, with `plugin.ts` from `master`: `npx vitest run --config vitest.config.ts src/plugin.test.ts -t "provider error state"` exits 1. All 7 new tests fail. The main case fails with `Sandbox failed to stop with status: error`, which is the production error. - SDK, with `worker-rpc-host.ts` from `master`: `npx vitest run tests/worker-rpc-host.test.ts -t "typed stop failure"` exits 1 (the typed case fails). - Server, with `recovery.ts` from `master`: `npx vitest run src/__tests__/heartbeat-pending-cleanup-sweep.test.ts -t "terminal provider stop failure"` exits 1. All 3 cases fail (`errorKind: "destroy_failed"`, no escalation). - Green on this branch: - `packages/plugins/sandbox-providers/daytona`: `npx vitest run --config vitest.config.ts` exits 0 (534 passed, 26 skipped live tests). `pnpm typecheck` exits 0. `pnpm build` exits 0. - `packages/plugins/sdk`: `npx vitest run` exits 0 (222 passed). `pnpm --filter @paperclipai/plugin-sdk typecheck` exits 0. - `server`: 15 related suites exit 0 (1498 passed): pending cleanup sweep, `heartbeat/recovery`, environment runtime, environment cleanup work, native startup cleanup, native remote failure retention, observed workspace preservation, retained native session cleanup, idle sleep safety, explicit native continuation, task retry, workspace recovery, native failed and cancelled workspace preservation, and Daytona close in the plugin worker manager. - `pnpm --filter @paperclipai/server typecheck` exits 0. `pnpm check:module-boundaries` exits 0. - New tests: - Daytona: a recoverable errored sandbox is recovered once (`recover(20)`) after the retention check and before Stop, then stopped. An unrecoverable one returns `resource_unrecoverable` with no Recover, Stop, start, delete or archive call. A failed recovery makes exactly one Recover call for each outcome (`recovery_failed`, `resource_unrecoverable`, `recovery_incomplete`). A Recover call that never settles is cut off by the bound. A recovered sandbox that fails again before Stop gives `recovery_failed` with no second Recover. - SDK: the envelope holds only the reason. A forged or non-SDK error gives no data. Unknown reasons, accessor fields and a sender-supplied `terminal` flag are not trusted. The worker host forwards the typed failure and drops a plain error's detail. - Server: terminal reasons escalate on attempt 1 with the long retry and one operator warning. `recovery_incomplete` keeps the short retry. The lease stays `pending_cleanup`, and the provider message never reaches the logs. - Not run: the full `pnpm -r typecheck`, `pnpm test:run`, `pnpm build` and `pnpm run typecheck:build-gaps` repo checks. CI runs them. No live Daytona probe ran. The mocks model the Daytona SDK `recover()` contract: it waits for `started` and rejects when the sandbox returns to `error`. ## Risks - Time budget: the host gives `environmentStopLease` at most 60 s. A recovered stop uses up to 20 s for Recover and up to 30 s for Stop, plus short reads. A slower recovery returns `recovery_incomplete`. The SDK call can continue at Daytona after that, so the next sweep finds the sandbox started and stops it normally. If the host deadline fires while the worker still stops the sandbox, the next sweep sees `stopped` and records the receipt. - Classification: after a failed Recover, the plugin cannot tell a rejected request from a sandbox that started and failed again. Both give `recovery_failed` (terminal) when Daytona still shows `error`. A hung Recover call with a sandbox still in `error` is also terminal. Terminal failures still retry every 30 minutes, and each retry makes at most one Recover call. An explicit Retry also makes one attempt. There is no tight loop. A sandbox that fails Recover every time gets about two Recover calls an hour until an operator acts. - Recover restarts the sandbox container before Stop. The plugin runs no command in it, and the retention policy is confirmed before Recover. - The continuation message for a terminally failed lease still says the message "will start automatically". This PR does not change admission. A follow-up can use the typed reason there. - Compatibility: a host without this change ignores the new error data and keeps the old retry behavior. A plugin without this change sends no typed data. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, code execution for tests and typechecks). The runtime does not expose the context window size to the agent. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (affected suites; full repo runs are left to CI, as noted above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-04 21:33:48 -08:00
DottaandClaude Opus 5.5	a994a648f1	fix(workspaces): name copy-back failures on the run and end failed save operations (#19187 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native run in a Daytona sandbox saves its workspace after the provider turn. This save is "copy-back". > - On staging, about 20 runs kept a saved result, but copy-back failed three times. The issues became blocked with `native_workspace_sync_out_retry_exhausted`. > - #19152 added a redacted cause for a sandbox command that exits with an error. Three gaps stayed open: a provider that refuses the command still gives a bare code, the run does not show the cause, and the failed save operation stays "running". > - The first failure also had the wrong name. The runner kept the result but did not prove a durable suspension. The executor reports this as `native_session_lease_lost`, and the workspace operation shows only that text. > - This pull request names each of these failures with closed, redacted labels. It also ends a failed save operation as "failed". > - The benefit is that operators can see why copy-back failed and which save is still active. Retries and saved results do not change. ## Linked Issues or Issue Description Refs #19152, #19145. These are the earlier copy-back diagnostics and retry-scheduling repairs. This PR does not duplicate them. What happened? A native task ran in a Daytona sandbox and its result was accepted. The run then failed with `native_workspace_sync_out_retry_exhausted`. The run had no error text. The automatic copy-back retries each failed in under one second, with only `workspace_sync_out_failed`. The final `remote_workspace_snapshot` operation stayed "running" after the retries stopped. The first `workspace_finalize` operation showed only `native_session_lease_lost`. The run log showed that the runner shutdown timed out before durable suspension. The lease was not lost. Steps to reproduce 1. Run a native task in a Daytona sandbox until its result is accepted. 2. Make the sandbox provider refuse the copy-back command (for example, a lease that is no longer active), so that no exit status comes back. 3. Let the automatic retries stop. Then read the run and its workspace operations. Expected behavior The run, the coordinator, and the failed operations name a redacted cause. The failed save operation is "failed" with a finish time. Actual behavior The run has no cause. The save operation stays "running". A provider refusal keeps only the bare `workspace_sync_out_failed` code. ## What Changed - `remote-workspace-transport.ts`: the provider can throw instead of returning an exit status for the copy-back helper, the capture, or the archive download. `remoteWorkspaceFailureCause` then returns `remote_workspace_operation_failed: operation=<op> transport=<label>[ code=<ERRNO>]`. The label is from a closed list (for example `sandbox_lease_inactive`, `permission_denied`, `sandbox_not_running`, `transport_timeout`, `transport_closed`, `daytona_sandbox_not_found`, `provider_error`). Provider text is never copied into the cause. The thrown error does not change, so preparation errors, `classifyNativeWorkspaceFailure`, and other callers keep their contracts. Archive-safety and Git-metadata refusals get no label, because they have their own diagnostics. - `remote-workspaces.ts`: when the last active invocation of a save fails, the save operation becomes "failed" with `exitCode`, `finishedAt`, a redacted `stderrExcerpt`, and `metadata.lastFailure`. A retry of the same boundary sets it to "running" again. A success sets it to "succeeded". After the final save commits, a failed save is retired for export cleanup in the same way as a running save. - `remote-workspace-snapshot-retention.ts`: a failed save that is not retired keeps its snapshot pinned, as it did while it was "running". Its retry can still use an uploaded snapshot. - `native-run-finalizer.ts` and `native-workspace-finalizer.ts`: the redacted cause of each copy-back attempt goes to the coordinator `failureDetail.failureCause` and the run `resultJson.failureCause`. When retries stop, the run error adds `Last cause: <cause>`. The server checks the cause against a fixed grammar again before it writes it. A cause with a path or other text is dropped. - `native-post-result-failure.ts` (new), `native-session-executor.ts`, `runtime-execution.ts`: a failure after an accepted result gets the label `runner_settlement_incomplete` when the runner did not prove a durable suspension. The failed `workspace_finalize` operation records this closed diagnostic as `failureDiagnostic`. The coordinator records it as `native_post_result_failure: category=<label>`. The control code `native_session_lease_lost` and its fencing do not change. ## Verification - Red, before the fix (the new tests run against `master` source): - 4 provider-refusal cases and the archive download failure: no redacted cause was available, so the copy-back recorded only the bare code. - Failed save, with only the transport fix applied: `- "status": "failed"` / `+ "status": "running"`. - Run cause after retries stop: `expected null to be 'remote_workspace_operation_failed: …'`. - Green, after the fix: `cd server && npx vitest run src/services/remote-workspace-transport.test.ts src/services/remote-workspace-save-concurrency.test.ts src/__tests__/native-workspace-finalization-work.test.ts src/services/native-runtime/native-post-result-failure.test.ts src/services/remote-workspaces.integration.test.ts src/services/remote-workspace-snapshot-retention.test.ts` passes. - Safety cases: - The failed save is retried on the same boundary operation, records "succeeded", and keeps the saved file bytes. - `daytona_sandbox_not_found` is still classified as `workspace_sync_out_unrecoverable`. - An archive-safety refusal keeps its exact text. - A forged cause with a path never reaches the run or the coordinator. - A run-level timeout cause (`remote_workspace_operation_timed_out: capture (configured timeout 300000ms)`) reaches the run and the coordinator. - `heartbeat-native-remote-failure-retention.test.ts` passes: a preparation failure keeps its exact run error. - The existing integration tests for retired saves, poisoned reservations, duplicate invocations, and snapshot retention pass. They prove export cleanup and identity protection. - `tsc --noEmit` for `server` reports no errors. - Not done: a live Daytona copy-back. This PR adds diagnostics and corrects the operation state. It does not change when a retry runs. ## Risks - Thrown copy-back errors do not change. Only the recorded cause is new. The `daytona_sandbox_not_found` and archive-safety classifications are tested. - A failed save operation is now "failed" before the final save commits. Retirement, export cleanup, and snapshot pinning now also select these failed rows. They select only rows with `lastFailure` and no retirement receipt. - Privacy: each new cause is a closed label or a server-built code. The server checks it again with an anchored grammar before it writes the run. No sandbox output, path, or message text is stored. - The retry count, the retry delay, and the classification of each failure do not change. No provider turn is added. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Provider: Anthropic. Model: Claude Opus 5.5 (`claude-opus-5-5`), in the Claude Code agent harness. - Capabilities used: tool use, shell and code execution, repository search, read-only browser reads of staging, and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-04 23:07:42 -06:00
Nicky Leach	53937be283	fix(agents): select environment plugins only for agents on that environment (#19173 )	2026-11-04 20:59:02 -08:00
Nicky Leach	dc46b575ac	fix(plugins): deliver agent runtime events only to the owning plugin (#19170 )	2026-11-04 20:57:19 -08:00
DottaandClaude Opus 5.5	512fbea0bb	fix(native): deliver messages saved while a native run finalizes (#19183 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A user can reply on a task while the agent's run is still finishing > - A remote native run first commits its result and its status decision (for example `done`). Then its sandbox environment settles (harness backup, warm retention). > - A message that arrives after the final turn took its input is saved as a deferred receipt behind that run > - The saved-message recovery path holds the receipt while the source run still owns its environment lease or its workspace copy-back owner > - Nothing notified that path when the lease or the owner settled, so the messages waited for an unrelated company event > - This pull request notifies the existing deferred-receipt work when a run's environment lease settles or its workspace finalization owner is released > - The benefit is that messages sent while a run finalizes are delivered once, soon after the run settles, and a task closed in that window is reopened for them ## Linked Issues or Issue Description What happened? A user posted three plain comments on a task while the agent's remote native run was finalizing. The final turn had already called its finish tool. The comments were saved in one deferred receipt. The run then committed `done` and its status became `succeeded`. No follow-up run started. The task stayed `done` with three unanswered requests. About 20 minutes later, an unrelated event in the same company woke the deferred-receipt worker, and the messages were delivered. Expected behavior A message that arrives after the last turn took its input, but before the run releases, gets one follow-up turn as soon as the run settles. If finalization marked the task `done` in that window, the task is reopened for the message. Messages that the finished turn already consumed are not replayed. Steps to reproduce 1. Start a native run with a remote (sandbox) environment. 2. After the agent calls its finish tool, post a user comment on the task. 3. Let finalization commit `done` and project the run as `succeeded`. 4. Observe that the comment's deferred receipt stays `deferred_issue_execution` after the environment lease settles. Root cause A remote native run is finished by native finalization, which projects the terminal status. The in-process run completion (`releaseIssueExecutionAndPromote`) does not run for it. Its saved messages depend on the stranded-queue sweep in `resumeDeferredExecutionReceipts`, which calls the release with `resumeStrandedQueue`. That release correctly waits while `hasPendingRunCleanup` is true (an unsettled environment lease or a workspace finalization owner). Finalization notifies the deferred-receipt worker when it projects the run, but the cleanup is still pending at that time. The worker then drops the entry. Lease settlement and owner release did not notify the worker again. Refs #19122 (saves a redirect sent while a stopped run settles; this change reuses the same deferred-receipt delivery path for normally completing runs). ## What Changed - `environmentService.releaseLease` notifies deferred-execution work for the company when a run's lease leaves `active` or `pending_cleanup` (released, retained, or failed). - `environmentService.releaseLeasesForRun` notifies deferred-execution work in the same way. - `withNativeWorkspaceFinalizationOwnership` notifies deferred-execution work when it releases its owner. - Delivery uses the existing path: `resumeDeferredExecutionReceipts` → stranded-queue release → deferred wake promotion (with the existing reopen rule for user messages). No new delivery mechanism. - New regression in `deferred-execution-receipt-work.test.ts` with the real deferred-receipt worker, the real release adapter, and the real lease service. - `doc/execution-semantics.md`: one additive paragraph for messages sent while a run finalizes. ## Verification - Staging evidence (read-only): the task's saved queue held the three comments as one `deferred_issue_execution` receipt after the run succeeded. The run had no `run succeeded` lifecycle event (it was finished by native finalization, not run completion). The receipt was promoted about 20 minutes later with `reopenedFrom: done` by the same recovery path. - Red before the fix: the new test failed with `expected null to be truthy` (the receipt stayed deferred with no run after the lease settled). - Green after the fix: `cd server && npx vitest run src/__tests__/deferred-execution-receipt-work.test.ts -t "finishing native run"`. - Safety cases in the new test: - The receipt stays deferred while the environment lease is active (cleanup holds still apply). - After settlement, exactly one run carries both messages in order, and the done task is reopened once. - A second settlement signal does not deliver the messages again. - Related suites: `deferred-execution-receipt-work`, `environment-service`, `environment-cleanup-work`, `native-environment-settlement-work`, `idle-sleep-safety`. - Not done: a live staging reproduction at the fixed revision. Deployment is out of scope. ## Risks - The deferred-receipt worker reloads the company's deferred receipts after each run lease settlement. Each reload is bounded and runs only when deferred receipts exist. The notification is transactional, so a rolled-back release does not wake the worker. - Low risk: no schema change, no API change, and no change to admission or reopen rules. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file edits, browser for read-only staging diagnostics). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-04 22:26:42 -06:00
DottaandPaperclip	a0250284e3	fix: preserve proposal revision turns and prove chat completion (#19105 ) ## Thinking Path > - Paperclip manages work done by AI agents. > - A completed delegated task must report its result to the originating chat. > - That report uses a real provider turn and must stay in cost accounting. > - Ordinary first-task proposals must also support the promised chat acceptance path. > - A revision response can reject an older proposal before it creates the replacement. > - The old rejection transition clears that active response's ownership and blocks its next tool. > - This change proves required completion work, guides ordinary proposal policy and preserves the revision turn under the existing locks. ## Linked Issues or Issue Description No separate issue exists. The original failures remain in [the complete Product run](https://github.com/paperclipai/paperclip/actions/runs/41915965118). The independent continuation repair is #19076. Related #12209 concerns prior-run confirmation binding. This change fixes the ownership reset during rejection, rather than changing binding authorization. What happened? The complete 20-case run finished with 13 passes and 7 failures. Three delegated-chat cases counted one required `chat_task_completed` provider turn as unexpected work. Two first-task agents chose optional human-only cards, then correctly received 403 when they tried conversational acceptance. A [reviewed two-case candidate run](https://github.com/paperclipai/paperclip/actions/runs/41931445311) then used the changed first-task skill. Both models inherited the ordinary resolver policy. Clarify/propose/accept passed all 12 predicates. Revise/accept failed after the new response rejected its predecessor's proposal. The saved rejection was followed by `paperclip_runner_tool_binding_not_authorized` on its next tools. The final `native_session_lease_lost` label followed those binding failures; this PR does not assume lease expiry caused them. Both cells cleaned up and published their original grades. Expected behavior Keep the existing ordinary-turn bounds and account for one independently proved completion notification. Inherit the normal resolver policy for ordinary proposals that accept chat replies. When an active response rejects an older proposal, save the rejection and retain that response's current ownership so it can create the revised proposal. Keep human, current-source and native completion-review rejection behavior and all real approval restrictions. Steps to reproduce The original full run used `4740d251`. The two-case candidate used trusted workflow `7d1e9b34` and target `74da56c4`. Run the focused provider-free controls below. The dual-run API regression first fails because rejection changes the issue to `todo` and clears the newer run's execution and checkout fields. ## What Changed - Prove one completion notification only for native Codex/local `multi-repository`, `clarify-reuse` and `plan-handoff`. Keep the ordinary bounds of 2, 3 and 4. Preserve actual totals of 3, 4 and 5, all run IDs and all usage. - Read those declared totals from the current catalog, which now includes the completion reply. Check the total once and prove that exactly one turn is the required notification. Do not add another turn while waiting. The required-server controls use the actual catalog tasks. - Bind the notification to the same company, agent, conversation generation, completed child, single delivery, accounted native run, saved reply and rendered result link. Reject extra work, duplicate deliveries, unknown events and mutations, including native work without a Paperclip tool call. - Collect workspace operations through the real run API for these three proofs. Keep other collector calls unchanged. Keep the proof controls in the existing required server test lane. - Stop idle polling on a permanent completion-proof error. Continue to retry transient observations within the original 240-second window. - Guide ordinary chat-accepting first-task proposals to omit optional `resolverPolicy` and inherit server/company defaults. Preserve explicit and governed human-only restrictions. Do not lower or recreate saved restricted cards. - Under the existing locked issue and interaction rows, preserve the active responding run when it rejects a different source run's proposal. Change only the ordinary requeue predicate. Keep the saved rejection, notifications, activity and existing Telemetry calls. Keep global status, lease and tool-authority guards unchanged. - Extend the existing native API controls with the actual dual-run rejection/revision sequence and policy, provenance, human, current-source and native completion-review countercontrols. No new workflow or test framework is needed. ## Verification - Current combined head is `a8567ded67e5ab811675390e9a5c5ed2194aee2f`, integrated cleanly on `fb96537e28c4c7bfced4939909cb334392320868`. The upstream completion-reply waiter is preserved. [Current required CI](https://github.com/paperclipai/paperclip/actions/runs/41940780852) passes. Actual test checkout `ff93c034d7bb40f3df7bc75cf51828fd07bb0d29` merges this head into `9c6993f0aab7ebb22a26d78f5d227accb35a6ad1`. - Fresh Greptile is 5/5 on exact head `a8567ded`, with no blocking finding. All addressed review threads are resolved. The separate automated reviewer emitted a paused-work response; that response is not a review approval. - Current required server CI actually passed all 11 native confirmation API cases, 46 completion-proof controls and 13 onboarding asset checks. Canonical typecheck, Build, the native package/server suites, all browser and serialized shards, `ci / e2e` and `ci / verify` pass. No current-head CI retry was needed. - Local dual-run RED: the unmodified transition returned HTTP 200 and saved rejection, then changed the issue to `todo` and cleared the newer run's execution/checkout fields. Whole setup and check took 28.95 seconds; the case took 596 ms. - Local treatment: all 11 real native/API cases pass. The same-session response retains execution/checkout ownership, reads task context and persists the recorded-shape revised confirmation. Its stored lease receipt stays intact and one rejection activity is saved. Human-only rejection, current-source rejection, human rejection with/without a newer owner, Plan/Ask, foreign provenance, authentication and formal approval controls pass. - All 59 existing confirmation resolver controls pass in 16.62 seconds. These cover cancellation, lost ownership, company/user scope, policy changes, transactional rollback, concurrent retries and stale revisions. - All 46 completion controls pass in 4.07 seconds. The two added clock controls preserve the original idle deadline and prove immediate permanent-proof failure plus transient observation retry. The existing live collector-to-proof and native mutation negatives remain in this required server file. - The first current-catalog integration check failed 5 controls in 2.70 seconds. The old consumer added the already-declared reply again and could accept an extra ordinary turn. The corrected consumer passes all 46 controls using the actual catalog. The existing full 20-case catalog/reporting compatibility check also passes in 4.93 seconds. Case identities, configured models and reporting match the declared 20. - The API controls used one owned loopback PostgreSQL instance through existing supported test hooks. It was stopped and removed. Earlier missing-library/setup failures and one corrected current-source test expectation remain retained; they are not counted as product failures or passes. - The unchanged first-task skill has SHA-256 `b789090996e7925676590c006418eb8b57f4514e646f6bca18ffd7b2b8e864de`. Both candidate runtime copies matched it. The inherited policy was observed in both models; one complete acceptance journey passed. This does not qualify the new rejection repair. - The three retained real completion streams pass the stronger local proof, including native items, workspace observations and recorded DOM links. Their original failed grades stay unchanged. - Earlier related checks passed 153 hiring/non-execution controls and 13 asset checks. Their source is unchanged. Required discovery includes the completion, API and asset files exactly once. - Prior candidate `74da56c4` passed all required CI, including canonical typecheck and build, in [CI run 41929148958](https://github.com/paperclipai/paperclip/actions/runs/41929148958). Its fresh Greptile was 5/5. This is prior-head evidence. - The intermediate `053150d9` head also received Greptile 5/5. Its [CI run](https://github.com/paperclipai/paperclip/actions/runs/41940014165) and review stay retained as superseded evidence. They do not qualify the final catalog correction. - A clean full local typecheck/build is not claimed because the reused dependency cache has stale adapter-utils/Runner declarations. No source or lock workaround was made. Fresh canonical CI must pass before merge. - The original full-plan result remains 13/20. The separate two-case probe remains 1/2. A fresh complete saved-plan run against delivered source is required for acceptance. No case, assertion, model, budget, timeout or failed grade is changed. Focused commands: ```sh # From server/; database URLs point only to an owned disposable local instance. node ../node_modules/vitest/vitest.mjs run src/__tests__/chat-completion-proof.test.ts PAPERCLIP_TEST_DATABASE_URL="$OWNED_TEST_DATABASE_URL" node ../node_modules/vitest/vitest.mjs run src/services/native-runtime/confirmation-comment-api.integration.test.ts PAPERCLIP_TEST_POSTGRES_ADMIN_URL="$OWNED_TEST_DATABASE_URL" node ../node_modules/vitest/vitest.mjs run src/__tests__/confirmation-comment-resolution.test.ts ``` ## Risks - The ownership exception applies only to an agent acting as the current execution run against a different non-null source run. Human, unowned and current-source requeue behavior must stay intact. Native completion-review handling remains separate. - The completion proof fails closed on new native tools or event shapes. A new shape needs review. - A stored unchanged lease receipt is supporting local evidence. The final real journey must prove full revision, execution and delivery. - The new head still needs normal protected merge approval. The optional queued Paperclip review is not a passing review. No causal repair of unrelated CI intermittency is claimed. ## Model Used OpenAI `gpt-6.1-sol` in Codex, with `max` reasoning effort. The runtime reports a 475,000-token context window. Used reasoning, code editing, shell tools and provider-free controls. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-04 22:04:12 -06:00
DottaandClaude Opus 5.5	06826915d7	fix(dispatch): park a workspace-busy wait instead of recreating runs (#19143 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Child tasks can reuse their parent's execution workspace. Only one run at a time may write in a shared sandbox workspace. > - A run that finds the workspace held by another live run waits with a `workspace_busy` scheduled retry. > - That wait cancelled the run and created a new retry run every 60–120 seconds, until the holder ended. Each check made one cancelled run record. > - Three sibling tasks that waited for a parent made 15 cancelled runs in 13 minutes. They also got the workspace in random order, not in arrival order. > - This pull request keeps the waiting run as one parked retry while its holder is live, and wakes it when the holder ends. > - The benefit is a clean run history, a prompt handoff, and fair arrival-order access to a shared workspace. ## Linked Issues or Issue Description Refs #18691 (the workspace wait for a live holder). Refs #19075 (sends a stopped writer with a pending save into the same wait). Refs #12587 (an earlier proposal about fairness for this retry). What happened? A parent task delegated three subtasks that inherit its execution workspace. Each subtask run was created, then cancelled with `workspace_busy` ("Shared execution workspace is busy: run … is still running"), then created again about every 1.5–2 minutes. One subtask got 8 cancelled runs, one got 5, and one got 2. Every cancellation named a holder that was really running: first the parent run, then the sibling that got the workspace. So serialization was correct. The cost was the churn of run records, and the subtask that waited first ran last. Expected behavior A run that needs a busy shared workspace waits as one durable record. It continues when the holder finishes. Siblings that wait for one workspace get it one at a time, in arrival order. Two writers never share the workspace, and the user can still stop a waiting run. Steps to reproduce 1. Bind two or more issues to one shared execution workspace in a sandbox environment. 2. Keep a run live on one issue (the holder). 3. Wake the other issues. Before this change, each due check promotes the retry, cancels it with `workspace_busy`, and creates a new retry run. After this change, the retry stays parked until the holder ends. Paperclip version or commit Observed on staging at `53735db7`. The code is unchanged on `master`. ## What Changed - `finalizeWorkspaceBusyDeferral` records `contextSnapshot.workspaceBusyWait` = `{ key, holderRunId, waitingSince }`. The key is `execution:<id>` or `project:<id>`. `waitingSince` carries over when the run defers again for the same workspace. - New `server/src/modules/run-dispatch/adapters/workspace-busy-wait.ts`: - `parkWorkspaceBusyRetryInTx` runs in the scheduled-retry promotion transaction, after the existing gate allows the retry. If the recorded holder is still running and is not stale, or an earlier waiter for the same workspace is still scheduled, queued, or running, the retry stays `scheduled_retry`. Only its deadline moves. It does not create a run, an event, or a wake. The fallback deadline is 5 minutes, and never later than the holder's one-hour stale point. - `wakeWorkspaceBusyWaitersInTx` runs in the transaction that ends a running run (`writeRunStatus`). It makes due the retries that wait for that run, and the retries that wait for the same workspace as that run. It registers the scheduled-retry dispatch notification before the write. - `WORKSPACE_BUSY_HOLDER_STALE_AFTER_MS` moves to the run-dispatch domain, so the deferral gate and the parked check use one value. `heartbeat/retries.ts` exports it again. - A retry without a recorded wait (for example, a retry from before this change) keeps the old behavior. - `doc/execution-semantics.md` describes the parked wait. ## Verification - Red before the fix, green after (`server/src/__tests__/heartbeat-workspace-busy.test.ts`, "reused shared execution workspace in a sandbox"): - "waits on a live holder in one parked run, then runs once when the holder ends": before, the retry was promoted while the holder was live. After, it stays parked through due checks at +1 s, +10 min, and +30 min. Ending the holder makes it due at once. It runs exactly once, and there is only one `workspace_busy` cancellation. - "hands the workspace to three waiting siblings in arrival order, one writer at a time": before, nothing woke the waiters when the holder ended. After, exactly one waiter is admitted per handoff, in arrival order. Each runs once, and there are 3 cancellations in total (one first deferral for each sibling). - "still lets the user stop a parked waiter": before, the waiter was promoted while the holder was live. After, Stop cancels the parked retry, and nothing brings it back. - Safety case (green before and after): "lets a stale holder reach the existing quiet-holder failure instead of parking forever". - `npx vitest run src/__tests__/heartbeat-workspace-busy.test.ts`: 35 passed. - `heartbeat-remote-workspace-busy`, `heartbeat-retry-scheduling`, `src/modules/run-dispatch`, `heartbeat/retries.test.ts`, `execution-projection.test.ts`: 266 of 267 passed. The one failure, "records pre-provider quota rejection…", is a timing flake on a loaded machine and passes when it runs alone. It does not use this path. - `tsc --noEmit` reports no errors in the changed files. - Staging evidence was read only. This change is not deployed. ## Risks - One extra indexed `SELECT … FOR UPDATE` on `scheduled_retry` rows when a running run ends. The update and the notification happen only when waiters exist. - If an earlier waiter stays queued for a long time (for example, its agent is paused), later waiters for the same workspace wait behind it. Each check is a deadline update, not a run. The fallback check runs every 5 minutes. - A remote writer that has ended but has not finished its save (#19075) is not parked, because its run is no longer `running`. Such a wait can still make a small number of deferrals until the save settles. - Low migration risk: no schema change. Old retries without the new context key keep the old behavior. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell, file edits, browser read of staging run records) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-04 22:01:37 -06:00
Dotta Paperclip Claude Opus 5.5	19c710fe93	fix(issues): record an interrupt comment as the user's Stop (#19149 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A board user can interrupt a running agent with a comment (`interrupt: true`) to redirect the work > - An interrupt is a Stop plus a message, but it did not record the user's stop identity the way the Stop control does > - The release path therefore treated an interrupted native run as a failed run: it blocked the task and took the finalization lease from the runner that was still settling > - Workspace preservation then stopped the sandbox under that live runner, so the runner reconnected for about 6 minutes and the stop could not be verified afterward > - The redirect message stayed saved, but it waited on workspace repair that could not finish > - This pull request records the board user's stop identity on interrupts, the same as the Stop control and terminalization stops > - The benefit is that an interrupt now settles like a Stop and the redirect can continue ## Linked Issues or Issue Description Refs #19122 (the Stop-then-redirect fix for the dispatch gate; this PR covers the interrupt-comment path). What happened? A board user posted a comment with `interrupt: true` on a task with a running native run (Daytona sandbox). The run was cancelled with `operator_interrupted` and the native cancel was acknowledged. Within a second, the task went to `blocked` and a `native_continuation_requires_reconciliation` hold was created. The runner transport then logged `state=reconnecting` for about 6 minutes, and the run ended with `runner_process_termination_unconfirmed: ... Daytona sandbox lease ... is no longer active`. Workspace recovery started about 10–17 minutes later and failed with `agent_stop_unconfirmed` / `provider_unavailable`. The redirect comment stayed saved as a deferred wake waiting on `workspace_repair_required`. No follow-up run started. Expected behavior An interrupt behaves like Stop followed by a message. The stopped runner settles normally. The task is not blocked as a failure. The saved message continues once the previous execution has stopped. Steps to reproduce 1. Start a native run on a remote sandbox. 2. Post a board comment with `interrupt: true`. 3. Observe the task become `blocked` with a `native-continuation:<runId>` hold, and the run's finalization row rewritten to `terminal_failure` with no lease owner while the executor is still settling. 4. Observe workspace preservation stop the sandbox, the runner transport reconnect until its grace ends, and recovery fail to confirm the stop. Root cause `recordNativeTerminalRecoveryIfNeeded` (`server/src/modules/wake-queue/adapters/postgres.ts`) skips intentional stops through `isAcknowledgedNativeStop`. That check requires `resultJson.cancelledByActorType === "user"` and `cancelledByUserId`. The Stop control and terminalization stops record these fields. `operatorInterruptCancelOptions` (`server/src/routes/issues.ts`) recorded only `interruptedByActorType` / `interruptedByActorId`. So an acknowledged interrupt took the failure branch. That branch blocks the task and sets `nativeRunFinalizations` to `terminal_failure` with `leaseOwner: null`. Cancelled-workspace preservation then saw a terminal row with no owner and stopped the sandbox while the runner was still settling. Paperclip version or commit Seen on staging at `69dea878c0`. The touched code is unchanged on current `master`. Agent adapter(s) involved Not adapter-specific (core bug). Seen with the Paperclip runner adapter on a Daytona sandbox. ## What Changed - `operatorInterruptCancelOptions` now records `cancelledByActorType: "user"` and `cancelledByUserId` for a board user, in addition to the existing interrupt fields. All three interrupt call sites (comment POST, issue PATCH, queued-comment interrupt) use it. - Route regression in `issue-update-comment-wakeup-routes.test.ts`: an `interrupt: true` comment on a running native run sends these fields, and the route's actual fields satisfy `isAcknowledgedNativeStop` with an acknowledged native stop receipt. - Release-path regression in `wake-queue/adapters/postgres.test.ts`: an acknowledged interrupt Stop, while the executor still owns finalization, releases the task without blocking it, creates no recovery hold, keeps the finalization lease owner, and keeps the saved redirect deferred. ## Verification - Red before the fix: - The route test failed because `cancelRun` received no `cancelledByActorType` / `cancelledByUserId`. - The release-path test failed with the old interrupt fields: the task was blocked, a hold was created, and the finalization owner was cleared. - Green after the fix: - `cd server && npx vitest run src/__tests__/issue-update-comment-wakeup-routes.test.ts src/modules/wake-queue` - Related suites: `acknowledged-native-stop`, `issue-queued-comments-routes`, `issue-comment-reopen-routes`, `issue-execution-policy-routes`, `explicit-native-continuation`, `heartbeat-process-recovery`. - The staging incident was read through the read-only run, event, log, recovery-action and wake diagnostics APIs. - Not done: a live staging reproduction at the fixed revision. Deployment is out of scope. ## Risks - An interrupt now counts as the user's Stop everywhere the Stop control does. This includes case-handoff supersession in `cancelRun`: an interrupt supersedes a pending case handoff, like Stop. - Only board users get the stop identity. Agent actors cannot use `interrupt: true` (the route returns 403). - Runs interrupted before this change keep their old fields. They are not migrated. - Not changed here, and suitable for follow-ups: - `recordNativeTerminalRecoveryIfNeeded` can still clear a live finalization `leaseOwner` for other failed native runs. - The runner PRP ingress reconnect loop (`server/src/realtime/runner-prp-outbound.ts`) does not stop early when the sandbox lease is closed. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file edits, subagent code tracing, browser for read-only staging diagnostics). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-04 21:58:51 -06:00
DottaandPaperclip	7c66b6b1f6	test(onboarding): cover plan, hiring, and execution approval history (#19069 ) ## Thinking Path > - Paperclip manages agent work and human decisions. > - Onboarding can approve a plan, a hire, and an execution step in separate turns. > - Tests covered single approvals more closely than that full history. > - The repaired gate must retain valid plan approval through later decisions. > - Every consumer must recognize the saved confirmation kind and selected scope. > - This change tests that contract and registers a complete interview-to-execution journey. ## Linked Issues or Issue Description Refs #19043, #17914, #18283. The earlier tests admitted one accepted proposal at a time. Decomposition fixtures used standard confirmations. The direct-task E2E path forbids hiring. Those cases did not cover an accepted checkbox plan followed by hiring and a separate untargeted execution approval. ## What Changed - Add eight PostgreSQL contract cases for standard and checkbox plan approvals. Native tools create the cards. An authenticated human accepts them through HTTP. A later native run reads the approved revision and actual selected scope. - Exercise native task creation, generic REST creation, child REST creation, and accepted-plan decomposition. Check the assigned child and one creation audit event. Check durable replay on the three entry points that support it. A pending execution confirmation still prevents creation. - Register the explicit-only `onboarding-approval-history` suite for local native Codex and ACPX Claude. The browser chooses the interview, selects only drafting in the saved plan, approves one writer hire, and approves a separate Start drafting confirmation. The model must hire the writer and use that writer's own child execution to save the welcome document. - Grade each human decision, the unchanged plan revision and selection, approval timing, one assigned child and worker execution, the exact writer-authored saved revision, no pending requests, and persistence after reload. Calibrate the oracle against missing, forged, premature, duplicate, and changed outcomes. - Preserve synthetic option choices, document/run bindings, and saved revision authorship in the existing public first-task evidence projection. Keep private provider/account data excluded. Document the workflow, evidence, billing, and cleanup bounds. - Align the approval-history fixture with its completion oracle: the onboarding parent must be Done. Test both native profiles against graded completion evidence and reject a completed child whose parent remains In review. Count parent heartbeats separately from the writer: accept one to seven parent records while retaining the twelve-company-run cap and the independent three-approval/one-writer oracle. - Retain the existing 52-cell first-task suite. The new suite has one attempt, seven expected work turns, a twelve-run maximum, a fifteen-minute deadline, and the existing $5 company task budget. It is excluded from `--all`. - Update two stale cleanup assertions to include the existing thirty-second deletion timeout. ## Verification - Red control: temporarily restore the pre-fix admission module and run the new checkbox/native history case. It fails with the expected `First-task proposal requires human acceptance` error after plan, hiring, and execution acceptance. Restore the repaired module after the probe. - The final post-rebase history matrix and existing onboarding safety suite passed together: 74 tests, no skips. - Final focused Runner E2E support passed: 413 Vitest tests and all 370 Node support checks. This covers catalog discovery, journey controls, wrong-outcome calibration, public evidence, and existing first-task behavior. - Review controls: four authorship cases fail before the revision check. A delayed browser-response control fails before the approval-time fix. Both pass after repair. The final oracle requires the exact current document revision, writer ID, native child run ID, matching body, and readback after reload. - Credential-free discovery lists both new native cells. No provider call was made. `OPENAI_API_KEY` and `ANTHROPIC_API_KEY` are absent, so the real provider journey remains unqualified. - The earlier broad credential-free E2E support run reported 6,728 passing cases, three skipped cases, and nine failures in three unrelated files. All nine reproduce with the changed E2E source restored to the unchanged baseline: three expired overhead-fixture cases, one stale Pi profile-version assertion, and five workflow security assertions. This change does not alter those files, paid workflow policy, or provider pins. - Previous full workspace typecheck/build and [Linux CI](https://github.com/paperclipai/paperclip/actions/runs/41921754638) passed on `4f431f5aae99d072cc75027e90d7b847b4c342ca`, including the new eight-case PostgreSQL file. Ponytail subsequently found a fixture/oracle terminal-status mismatch. Both native-cell controls fail on that fixture (`done` observed versus `in_review` expected), while the parent-in-review negative control passes. A second comparison uses the actual `matchesRunCount` predicate: graded evidence with six parent runs and one writer run fails the old exact-seven-parent requirement. Both fixes pass these controls, all 413 focused support tests, all 370 Node checks, and the owning E2E project typecheck. [Fresh CI](https://github.com/paperclipai/paperclip/actions/runs/41936699686) passed on `91c980b110252da451b92eda597c168517de1276`: 48 successful jobs and one expected skip. Greptile reviewed this head at 5/5 with no unresolved findings. Ponytail's terminal-status finding is fixed and its thread is resolved; its current-head re-review remains queued. Live provider cells remain unqualified. ## Risks The production API and approval gate are unchanged. The new paid journey can expose model or product failures at an intermediate approval boundary. It retains those failures and uses the existing diagnostics and cleanup. Registration and deterministic tests do not establish a live provider pass. The unrelated baseline support failures remain explicit. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools, and code execution. The exact serving model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (focused tests; unrelated baseline failures are recorded above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-04 21:46:55 -06:00
Devin FoleyandClaude Opus 5.5	b9fbbf8d30	fix(native): continue runs woken by withdrawn or cancelled interactions (#19163 ) Withdrawing a card or cancelling a question wakes the assignee with that interaction id. The native interaction bridge required every wake interaction to be accepted, rejected or answered, so preparation failed with native_interaction_unresolved before the provider started. Project a cancelled interaction as a closed response with its stored result. Consumers grant authority only for accepted, answered or pending items, so the closed response carries context and no approval. Pending requests and governed tool reviews keep their existing checks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-04 19:10:59 -08:00
Devin FoleyandClaude Opus 5.5	3c7af35103	fix(errors): return tool gateway rejections with their own status (#19158 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agents call connected tools through the tool gateway. Some tool calls need a human review before they run. > - A user approves a tool review from a task card. The approve route calls the gateway. > - The gateway can reject the approval with `ToolGatewayHttpError(409, "approved_tool_target_changed")`. This is an expected rejection: the tool or connection changed after the review. > - `ToolGatewayHttpError` does not extend `HttpError`. The shared error handler only keeps the status of an `HttpError`. Thus the route answers 500 "Internal server error" and sends the rejection to Sentry as a crash. > - The user does not see the reason. The user clicks Approve again, and each click makes one more crash report. One user made 69 reports on staging this way. > - This pull request makes the shared error handler keep the status and reason code of client (4xx) gateway errors. > - The benefit is a clear 409 with a reason for the user, and no false crash reports. ## Linked Issues or Issue Description Refs #12775 (related community PR. It changes `ToolGatewayHttpError` to extend `HttpError`, which affects every `instanceof HttpError` check. This PR is smaller and keeps gateway server failures on the crash path.) What happened? A user approves a tool review on a task card. The tool definition or connection changed after the review. The API answers 500 "Internal server error". Sentry records the expected rejection "Tool definition or connection changed; request a new review" as a server crash. Each retry records one more event. Expected behavior The API answers 409 with `code: "approved_tool_target_changed"` and the message "Tool definition or connection changed; request a new review". Sentry does not record a crash. Steps to reproduce 1. Make a connected tool that needs approval, and start a tool call so that a review card appears. 2. Change the tool connection (for example, its config) before you approve. 3. Click Approve on the task card (`POST /api/issues/:id/interactions/:interactionId/accept`). 4. See 500 "Internal server error" and a new Sentry event from `approveActionRequest`. Paperclip version or commit `master` at `482ec0aea9ab6f2684baaa1487f2c7a2eedf009e`. ## What Changed - `server/src/middleware/error-handler.ts`: the handler converts a `ToolGatewayHttpError` with a 4xx status to an `HttpError`. The new error keeps the status, the message, the gateway details, and the reason code as `code`. The original stack stays. - Gateway errors with a 5xx status do not change. They still go on the crash path, answer 500 "Internal server error", and go to Sentry. Thus provider text in those messages stays hidden. - `server/src/__tests__/error-handler-tool-gateway.test.ts`: new tests for the 409 mapping, for the unchanged 5xx path, and for an Express route that lets the gateway error escape. ## Verification - `cd server && npx vitest run src/__tests__/error-handler-tool-gateway.test.ts src/__tests__/error-handler.test.ts` passes: 21 tests. - Without the fix, the new tests fail with `expected 500 to be 409`. With the fix, they pass. - `src/__tests__/issue-thread-interaction-routes.test.ts` and `src/__tests__/json-parse-error-handler.test.ts` also pass: 175 tests in 4 suites with the two error handler suites. - I checked the runtime import graph of `services/tool-gateway.ts` (685 modules). It does not import `middleware/error-handler.ts`, so the new import makes no cycle. - I did not run the full server typecheck locally, because it ran out of memory in an 8 GB sandbox. CI must do this check. ## Risks - Low risk. Only gateway errors with a 4xx status change. Before, they became 500. Now they keep their own status. - Some routes outside the gateway router can now return gateway 4xx details in the `details` field. The gateway routes already return the same details through `sendGatewayError`. - `middleware/error-handler.ts` now imports `services/tool-gateway.ts`. Unit tests that import the error handler load more modules. The app already loads this module. - This PR does not fix the cause of some of these rejections. A routine OAuth token refresh changes values in the signed approval snapshot (token secret versions, the order of grant secret refs, and token expiry times). This can make a pending review fail. That fix changes `services/tool-gateway.ts` and comes in a separate PR. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used as a coding agent with tool use (shell, file edits, test runs, and GitHub API). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes (no documentation change is necessary) - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-04 18:14:52 -08:00
Devin FoleyandClaude Opus 5.5	7edbc1fc07	fix(server): keep a successor's issue lock when a finalizing native run releases its checkout (#19156 ) `release_checkout` now clears `checkoutRunId` and the execution lock only when they belong to the committing run, so a successor that claimed the issue while its predecessor was finalizing keeps its claim and passes reviewed-chat attestation. `attestReviewedExternalChatRun` reports the first failed check as a closed label. Setup failures store it in `resultJson.reviewedChatBinding.denialReason`, and Sentry run-failure diagnostics export it as `execution.reviewedChatBindingDenialReason` (no IDs or text). The error message is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-04 17:49:03 -08:00
DottaandClaude Opus 5.5	482ec0aea9	fix(native): run saved copy-back retries on schedule behind a slow export (#19145 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs copy their sandbox workspace back after the provider finishes. A failed copy-back saves a retry 30 seconds later, with a bound of 3 attempts. > - One shared delivery queue runs these retries. A pass of that queue ends only when its slowest check ends. > - A legitimate sandbox export can take 10+ minutes. While it runs, all other due retries wait. > - On staging, a run active across a deploy showed "Reconnecting" with an overdue retry for about 24 minutes. Its retries ran at +24 min and +3.5 min instead of +30 s. It settled only after about 32 minutes. > - This pull request lets idle check slots admit entries that come due during a pass. It does not overlap a running check's resource and does not start work after abort or drain. > - The benefit is that the existing 3-attempt bound settles a failing copy-back in about 1 minute, not after the slowest unrelated export. ## Linked Issues or Issue Description What happened? A native run whose result is durable but whose workspace copy-back failed stays in `running` with the label "Reconnecting" for a long time. Its saved retry time is in the past. The retry does not run until an unrelated slow copy-back in the same queue pass finishes. Steps to reproduce 1. Start two native runs on different tasks with accepted results. 2. Let the copy-back of run A take many minutes. 3. Let the copy-back of run B fail. It saves a retry 30 seconds later. 4. Watch run B. Its retry does not run until run A's copy-back ends. Expected behavior Run B retries on its saved schedule. After 3 failed attempts it settles with `native_workspace_sync_out_retry_exhausted` and a recovery action, while run A continues. Paperclip version or commit master at `a0c0851766`. Observed on a staging deployment of `69dea878`. Agent adapter(s) involved Not adapter-specific (core bug). Observed with the Paperclip runner. Relevant logs or output In-run copy-back failed at 23:44:57 (retry saved for 23:45:28). The deploy restarted the server at 23:49:45. One queue check exported another task's workspace from 23:53:54 to 00:06:15. The overdue retry ran at 00:09:06, then 00:12:39, and the run settled at 00:12:40 with `native_workspace_sync_out_retry_exhausted`. About 20 other runs followed the same timeline. Their failures at 00:12–00:13 are this same retry exhaustion. They do not come from a reattach sweep. ## What Changed - `registerScopedDeadlineWork` has a new opt-in `concurrentChecks.admitDueWhileActive` option. While a pass still has a running check, idle slots admit entries that are due (saved deadline, failure backoff, and minimum spacing). They wait for the next deadline or the next check completion. - Admission skips entries owned by the pass and entries whose resource keys a running check holds. It stops on abort or when `canRun()` is false. The pass still ends only when all running checks end. - The native workspace finalization queue opts in. Its `check` already rereads the durable coordinator row before it acts. - Regression tests: a real-database test in `native-workspace-finalization-work.test.ts`, plus two helper tests (resource-key safety and shutdown). ## Verification - `cd server && npx vitest run src/__tests__/native-workspace-finalization-work.test.ts src/__tests__/scoped-deadline-work.test.ts src/__tests__/delivery-work-coordinator.test.ts` passes (68 tests). - Red before the fix: "keeps another task's saved retries on schedule and settles it while a slow copy-back is running" failed because `onWorkspaceSettled` was never called for the failing run. "admits saved deadlines that come due behind a slow check without overlapping its resource" failed too. - Green after the fix: the failing run makes exactly 3 attempts, settles as `terminal_failure` / `native_workspace_sync_out_retry_exhausted`, and the slow copy-back is admitted once and then succeeds. - `tsc --noEmit` shows no errors in the changed files. - Not done: a live staging check. This change is not deployed. ## Risks - Low to medium. Only the workspace finalization queue opts in. Other queues keep the current behavior. - An admitted entry is not reconstructed first. The finalization `check` rereads durable state, so a stale in-memory entry cannot start a second export. The advisory lock and owner receipt still prevent two copy-backs for one run. - This change does not fix the cause of the copy-back failures on staging (`native_session_lease_lost` in-run, then `workspace_sync_out_failed` with no detail). That needs a separate change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-04 19:26:02 -06:00
Dotta Paperclip Devin Foley	c2cc69d1e4	fix(recovery): resume and settle preserved native results (#19034 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native execution can save an accepted result before workspace export fails. > - Recover must preserve the files and settle that same result without another agent turn. > - The export failure can own the task's Blocked transition and its recovery notice. > - Recovery must prove preservation, stop, source identity and current authority before settlement. > - Recover can also transfer repair ownership to the system, which must keep explicit board export repair available. > - This change restores those paths while preserving later task state, live owners and unknown action outcomes. ## Linked Issues or Issue Description Related #17916 protects unfinished exports. Related #19004 preserves deliberate Retry authority. This change repairs accepted-result finalization and its recovery notice. What happened? A native run accepted its result, then exhausted workspace export. Normal Recover could save the files but leave the result and old recovery notice unsettled. A saved Retry could also use a different generation from preservation. Separately, Recover changed the action owner from board to system. Explicit export Retry then rejected that same source as stale, even when its accepted result and stopped allocation still matched. Expected behavior Preserve files, finish the original accepted result and close only its obsolete notice after exact proof. Keep explicit board export repair available after system adoption. Preserve current authority, Stop, later task changes, newer execution and every physical ownership fence. Steps to reproduce 1. Accept a native result and exhaust its workspace export. 2. Request ordinary Recover, optionally with an existing saved Retry. 3. Let file preservation complete. Inspect the task and source recovery notice. 4. To reproduce the explicit-repair defect, call export Retry after Recover adopts the action. Before this repair it rejects the valid system-owned source before any provider call. ## What Changed - Record the exact accepted result, contract, task status version and prior decision when export exhaustion creates Blocked. - Bind passive finalization to complete, source-bound preservation and the current chat/review authority. Recheck the proof under status commit locks. - Keep the saved Retry identity separate from its preservation generation. Retain coalescing and reject malformed or mismatched history. - Preserve an existing published GitHub assessment for the no-publication status-unproven wait. Keep all semantic, result, authority and publication checks. - Close an old export notice only after its own accepted result, export or preservation receipts, effects, projection and old Runner ownership settle. Keep newer work and files unchanged. - Permit explicit board export Retry after Recover transfers the exact source to the system. Require valid system and workspace recovery records for that same run, plus every existing result, stop, allocation, task and shared physical-lock check. Reject missing proof, another source and agent ownership before provider access. - Merge current master while retaining the stronger Stop, source and verified preservation checks. - Update the execution contract. No schema, endpoint, Runner protocol, model, trust or provider artifact change. ## Verification - Current head: `9be5a73489801095c9f39c9bfcb285a22b055912`. - The explicit-repair regression fails before the new guard change: ordinary Recover adopts the real database action, then export Retry rejects it as stale. It passes after the repair. - On the merged source, three owning PostgreSQL/file suites pass 220 tests: `preserved-native-response-wait.integration.test.ts`, `native-workspace-export-retry.test.ts` and `idle-completed-history.test.ts`. The export suite includes four unrelated-owner/source rejection controls and both board-owned and system-adopted active-exporter controls. Results and run counts prove that repair creates no provider turn. - Server layered typecheck, module boundaries, `git diff --check`, and the added-line secret/private-incident scan pass. - Full workspace typecheck (`pnpm -r typecheck`) and build (`pnpm build`) pass. The complete local `pnpm test:run` is still running; its completed serialized chat partitions pass. Current-head CI is green (53 successful checks, 3 expected skips). Greptile reviewed `9be5a73489801095c9f39c9bfcb285a22b055912` at 2026-11-05 00:58 UTC with confidence 5/5; all review threads are resolved. - Original qualification remains supporting evidence: the accepted-result fixture exercises HTTP Recover, real files, the finalizer, saved Retry coalescing, eight commit races, current principal/resource/head changes, Stop, later task changes, notice closure and repeated delivery. No accepted response, provider turn or publication is replayed. - Earlier failures remain recorded. Original red controls exposed wrong-action preservation, unrelated finalization barriers, generation mismatch and a lost GitHub assessment; the original repair added controls for them. The prior CI run [41908572248](https://github.com/paperclipai/paperclip/actions/runs/41908572248) failed an unchanged completed-history runtime fixture because its loopback service refused connections through the original ten-second readiness window. That owning suite passes unchanged in the 220-case run above. The earlier readiness failure's cause remains unproven; no wait or assertion was weakened. - Local and CI tests do not establish hosted task recovery. The serving revision and original-task outcomes require separate verification after rollout. ## Risks - Historical failures without the exact export-owned Blocked receipt keep their current task status. A saved result does not grant execution or publication authority. - Explicit export repair still resumes only the same retained allocation under the shared physical lock. It cannot take over an active exporter, create another agent turn, choose a replacement sandbox or discard saved files. - Retiring a source notice does not release physical resources or modify a newer task owner, later Done state or saved snapshot history. - The proof adds bounded reads only for concrete recovery obligations. It adds no recurring worker or polling timer. ## Model Used OpenAI GPT-6 through Codex for the original repair. OpenAI GPT-6.1 Sol (`gpt-6.1-sol`), xhigh reasoning, for the merge, integration verification and system-adopted export repair. Both used repository tools, code execution and PostgreSQL tests. The runtime does not expose a context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used - [x] I have checked ROADMAP.md and searched for duplicate or related work - [x] I have described the issue and linked related public PRs - [x] I have included no internal task identifiers or private incident links - [x] I have added behavioral and safety regression tests - [x] I have documented verification limits and risks - [x] I have updated the execution contract - [x] All Paperclip CI gates are green on the current head - [x] Greptile is 5/5 on the current head with no unresolved comments - [x] I will address review comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Devin Foley <devin@paperclip.ing>	2026-11-04 17:16:17 -08:00
DottaandPaperclip	9e5039f6c8	fix(daytona): retain duplex close ownership after lease cleanup (#19150 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Remote agents use provider plugins to manage sandbox leases and duplex channels. > - The host verifies the exact route and session when a bound channel closes. > - Daytona lease teardown removed that binding before a later host close. > - The incomplete reply made the host retire the shared worker. A later command then failed during setup. > - This pull request retains bounded close receipts and joins the original cleanup. > - Later work can use the same worker while invalid or unconfirmed closes still fail closed. ## Linked Issues or Issue Description What happened? A normal lease teardown can close a duplex channel before the host closes its bound route. Daytona then replies with only the route ID. The host requires the original session ID too. It logs `duplex channel close not acknowledged; retiring worker`, stops the worker, and rejects a later `environmentExecute` call as stopped. Expected behavior A later close must identify the original route and session and join its existing cleanup. It must not revive the channel or restart the worker. Unknown ownership and an unconfirmed close must retain the existing failure behavior. Steps to reproduce 1. Acquire two leases through one Daytona provider worker. 2. Open a duplex channel on the first lease. 3. Release that lease, then close the bound channel through the host. 4. Execute a command on the second lease. Paperclip version or commit The real host/worker regression fails on `69dea878c0b920b335594debb0286443ba9f6c35`. This branch includes master `3a72efebe5b050e7594bb309e71a0d5f3072db00`. Deployment mode Managed Daytona sandbox execution in Cloud. The regression uses a local real host and SDK worker. Only the external Daytona SDK and PTY are simulated. Related work: Refs #19004, Refs #10429, Refs #18065, Refs #19050. These cover task Retry, managed worker startup, drain release, and stale provider sessions. ## What Changed - Retain at most 4096 retired route/session bindings with their original close promises. - Remove live channel authority before cleanup starts. Concurrent and later closes join the same promise and return the stored binding. - Reject reuse of a retained route ID. Keep unknown and evicted routes unable to provide a session acknowledgement. - Add a real child-process host/SDK-worker/Daytona regression with only the external Daytona SDK and PTY simulated. - Cover release, cancellation, stop, destroy, resume, a stalled close, concurrent closes, failed cleanup, forged IDs and bounded eviction. - Document close receipt scope and its separate provider-termination checks. ## Verification - RED before the repair: the actual host and SDK worker reproduced the retirement log, SIGKILL, stopped state, and later `environmentExecute` rejection. The direct provider cases also failed because the session ID was missing. - Combined-branch provider suite: 364 passed. - Combined-branch host manager, duplex, managed readiness and composed suites: 142 passed. The final test-only manifest-loading change passed all 6 composed cases again. - Full repository typecheck (`node scripts/typecheck-low-memory.mjs`), focused server types and the standalone Daytona build passed. The provider build uses pinned SDK `0.203.0` and TypeScript `5.9.3`. - The composed cases check the same worker/PID, one later command, preserved saved content, revoked channel writes/stops, and continued rejection of commands on closed leases. A stalled close still retires the worker and dispatches no later command. - Reproduce with `pnpm --dir server exec vitest run --config vitest.config.ts src/__tests__/plugin-worker-manager-daytona-close.test.ts` from a checkout with the SDK build dependencies prepared. Run the provider suite with its own Vitest config. - An initial local typecheck lacked the excluded providers' SDK links after an install with scripts disabled. The standard root postinstall linker restored those local links, and full typechecking then passed. The fixture loads its actual built manifest. Failed setup output is retained; no type configuration was relaxed. - The first two composed attempts stopped during fixture setup. They are not the behavioral RED above. Their failed outputs are retained. - The first broad local typecheck stopped at an IPC permission error during preflight. It passed after rerunning with allowed local IPC and the standard SDK links. - [Full repository CI](https://github.com/paperclipai/paperclip/actions/runs/41945022881) passed all 48 jobs on this head. Selection fell back to the complete task plan, so the run was unfiltered. The complete check inventory has 54 successes and two expected Storybook skips; the separate Snyk status passed. Fresh Greptile review on `daebbeb5005b9d9bd59d32ea30094279048b9213` is [5/5 with no actionable comments](https://github.com/paperclipai/paperclip/pull/19150#issuecomment-6646621914). Independent production and fixture review found no actionable findings. Hosted acceptance of this repair has not run. ## Risks - A receipt removed at the 4096-entry bound cannot confirm an old bound close. The existing host guard remains effective in that case. - Close acknowledgements identify channels. They do not certify remote process termination or clear unknown external work. Provider termination receipts and existing recovery gates remain in force. - Hosted evidence confirms the unacknowledged-close retirement class. The historical log does not distinguish a missing session ID from a timeout or another close error. The regression proves and repairs the missing-binding path; it also preserves timeout retirement. - The provider is outside the main pnpm workspace. A new Cloud image must compile its changed source. The unchanged plugin version alone does not prove activation. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, source editing and tool execution. The exact model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-04 19:15:50 -06:00
DottaandClaude Opus 5.5	d8dbe55368	fix(workspaces): let a missing task repository ref be corrected (#19072 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents can ask Paperclip to prepare a task repository at a specific ref > - Paperclip records that request and clones the repository at the next admission > - Paperclip did not check that the requested ref exists > - When the ref was absent, every admission retried the same checkout and failed setup > - A corrected request was refused as a conflicting receipt, so the task stayed blocked with no repair path > - This pull request identifies a missing ref and lets a corrected request replace it while nothing was published > - The benefit is that a bad ref becomes a repairable error instead of a permanent setup failure ## Linked Issues or Issue Description What happened? An agent asked for a catalog repository at ref `main`. The repository default branch is `master` and has no `main` branch. Each later admission failed with `setup_failed`: `git checkout main` reported `pathspec 'main' did not match`. The task failed admission for more than a week and became blocked. A new request with ref `master` returned 409 because the receipt with ref `main` already existed. Expected behavior Paperclip should report that the requested ref does not exist. A new request with a valid ref should replace the bad ref when no checkout exists. Steps to reproduce 1. Use a repository whose only branch is `master`. 2. Request it as a task repository with `ref: "main"`. 3. Admit the task. Preparation fails, and the receipt goes to `needs_repair`. 4. Request the same repository with `ref: "master"`. Before this change, the request returns 409. ## What Changed - `managed-repository-checkout.ts`: When `git checkout <ref>` fails with Git's unknown-ref message, and each quiet `rev-parse --verify` probe (`<ref>^{commit}`, `refs/remotes/origin/<ref>^{commit}`) exits 1, throw `RepositoryRefNotFoundError`. The message tells the caller to request the repository again with an existing branch, tag, or commit. Other checkout failures keep their original cause. - `execution-workspace-repositories.ts`: Record `failureCode: "repository_ref_not_found"` for that error. Other failures keep `repository_preparation_failed`. - `execution-workspace-repositories.ts`: A request for the same repository with a different ref replaces the ref only when all of these are true: the receipt is `needs_repair` with `repository_ref_not_found`, it has no pinned commit, the workspace is local (no remote authority), and the checkout path is absent. The update is conditional on that state and writes a `workspace.repository.ref_corrected` activity entry. All other ref changes keep the existing 409 conflict. - `execution-workspace-repositories.ts`: Local admission re-reads each receipt inside the checkout lock and claims it with a conditional update on its current ref. A correction that commits after the receipts were listed is used, not discarded. - Tests: one regression for the full missing-ref path, and one safety test that keeps the receipt immutable when files exist at the checkout path or when the failure was not a missing ref. ## Verification - Red before the fix: `pnpm --filter @paperclipai/server exec vitest run src/__tests__/task-workspace-repositories-persistence.test.ts` failed with `expected ... /ref "main" does not exist/ but got 'Failed to prepare managed checkout fo…'`. - Green after the fix: the same file passes (11 tests). The new test admits with ref `main` (fails, `repository_ref_not_found`, no checkout), corrects to `master`, admits again (ready, pinned to the source commit), and then confirms that a later ref change after publication still returns a conflict. - The regression also holds the checkout lock while admission waits, corrects the ref, and then releases the lock. Without the in-lock re-read it fails with 409; with it, admission publishes the corrected ref. - `heartbeat-managed-clone-credentials.test.ts` passes: an authentication failure during checkout keeps its cause and is not reported as a missing ref. - Other checkout suites pass (9 files, 317 tests): reopen, base storage, expiration, project env, rehome, nested clone migration, object cache, trust, source copy privacy. - Neighbor suites pass: `managed-repository-checkout.test.ts`, `task-workspace-repositories.test.ts`, `task-workspace-repositories-trust.test.ts`, `heartbeat-project-repositories.test.ts` (47 tests). In one parallel run under heavy machine load, `heartbeat-project-repositories.test.ts` had timing failures in different tests on each run. The same file passed alone with this change (47/47) and on master. - `tsc --noEmit` for `server` passes. ## Risks - Low risk. The new correction path is narrow and fails closed. Remote (sandbox) workspaces keep the existing conflict because the control plane cannot inspect the remote checkout. - Existing receipts that failed before this change have `repository_preparation_failed`. They need one more failed admission to get `repository_ref_not_found` before a correction is accepted. - No schema change. `failure_code` is free text. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, agentic coding with tool use (shell, file edit, browser read-only diagnostics). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-04 18:25:41 -06:00
Dotta Paperclip Claude Opus 5.5	35d086213e	fix(native): explain failed accepted results on the run (#19091 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native run ends when the server projects the accepted result onto the run row. > - Operators read `errorCode` and `error` on a failed run to learn why it failed. > - Three native writers can set a run to `failed` without these fields. #19063 repairs the finalizer retry-exhaustion writer. > - The two result-projection writers still set `failed` with no cause when the provider failure is not classified. > - This pull request makes both projection writers record a cause when they project `failed`. > - The benefit is that, with #19063, no native writer can leave a failed run unexplained. ## Linked Issues or Issue Description Refs #19063. What happened? A staging soak found 11 failed native runs with `errorCode` null and `error` null. In each run the provider turn succeeded. Workspace copy-back then failed three times, and the finalizer retry-exhaustion path set the run to `failed`. #19063 fixes that writer. An audit of every `heartbeat_runs` status writer found two more writers with the same defect. `finalizeNativeRun` and `projectCommittedRun` project an accepted result with `runTerminalState: "failed"`. They write the error fields only when a classified provider failure exists. Expected behavior A run with status `failed` always has an `errorCode` and an `error`. A cause that is already recorded stays. Steps to reproduce 1. Accept a native result with `runTerminalState: "failed"` and no `turn.failed` event. 2. Call `finalizeNativeRun` with `projectRunStatus: true`, or project the committed run through `projectCommittedRun`. 3. Read the run. On master its status is `failed`, and its `errorCode` and `error` are null. ## What Changed - Add `explainFailedRun` in `server/src/services/native-runtime/native-run-failure-explanation.ts`. It builds SQL SET expressions. They keep a non-empty existing `errorCode` or `error`. Otherwise they write `native_run_failed` and a readable message. By default the expressions leave a run that is already `cancelled` unchanged, as the status writers do. - `finalizeNativeRun` uses the helper when it projects `failed`. The existing provider-failure and recovered-failure values remain the preferred causes. - `projectCommittedRun` uses the helper for a `failed` accepted result. This writer sets status unconditionally, so it does not use the cancelled branch. - Writer audit: all legacy and heartbeat callers of `setRunStatus` that pass `failed` already set both fields. The other direct native writers also set both fields. ## Verification - Red, with the source change stashed: 3 new cases failed with `errorCode: null`. These were the finalizer projection, recovery projection with no cause, and recovery projection with an empty error. The control case with a recorded cause passed. - Green: `npx vitest run src/__tests__/native-run-projection-work.test.ts src/services/native-runtime/native-run-finalizer-telemetry.test.ts src/__tests__/native-status-arbiter-corpus.test.ts` (225 passed). - Wider run: all `server/src/__tests__/native-.test.ts` and `server/src/services/native-runtime/*.test.ts` (7416 passed). Two failures come from the local environment. `plugin-runner-probe` needs the Rust runner source. `native-ingress-bootstrap-retry` needs the controller-host OpenCode proxy. - This change was not exercised on staging. It changes only fields that are written at terminal projection. ## Risks - Low risk. The change affects only rows that are projected to `failed`. A recorded cause is never replaced. Succeeded and cancelled projections are unchanged. - The staging failures need #19063 too. This PR alone does not change their writer. - The cause behind the staging copy-back failures is still unknown. The sandbox command reports the redacted `remote_workspace_operation_failed`, which also appears in a separate signature. This PR does not change that diagnostic. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, file edits, browser reads of staging run records). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-04 17:40:05 -06:00
DottaandPaperclip	b7ef307a65	fix(workspaces): wait for an ended writer's save instead of failing setup (#19074 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agent runs can use a remote task workspace in a sandbox. One writer run at a time holds that workspace. > - When a writer run ends, its workspace save and sandbox release finish a few minutes later. The writer hold stays until then. > - A new run that started in that interval failed with `setup_failed`: "The workspace still has an unfinished writer or save operation." > - A live writer already makes the new run wait (`workspace_busy`, added in #18691). An ended writer that is still settling did not. > - This pull request reports the ended writer as the holder too, so the new run waits instead of failing. > - The benefit is that work continues automatically after a short settle interval. Users do not see a failed run and a blocked task. ## Linked Issues or Issue Description Refs #18691 What happened? A task run failed during workspace setup with `setup_failed` and the error "The workspace still has an unfinished writer or save operation." The previous run on the same task had ended a few seconds before. Its sandbox lease was released 4 to 17 minutes later. A staging soak recorded 31 of these failures over 14 tasks in about 29 hours. What did you expect to happen? The new run waits for the workspace, as it does when the previous writer is still running. It continues when the save and release settle. Steps to reproduce 1. Bind a task to a remote workspace whose authority has `dirty: true` and a `writerRunId` for a run with status `failed` that ended now. 2. Wake the task's agent. 3. The run fails with `setup_failed`. No retry is scheduled. ## What Changed - `reserveRemoteWorkspace` now throws `RemoteWorkspaceWriterActiveError` for an ended writer whose save or sandbox release has not settled. Before, it threw a plain 409 conflict. The error keeps the same message, status and `remote_workspace_writer_pending` code. It uses the writer's `finishedAt` as its last activity and sets `ended: true`. - Workspace preparation already converts that error to a `workspace_busy` deferral with the one-hour holder cutoff. It now passes `holderEnded` to the deferral. - `WorkspaceBusyDeferral` says "ended, but its workspace save has not settled" for an ended holder, instead of "is still running". - A writer that is missing or has no task still gets the plain conflict, as before. - `doc/execution-semantics.md` describes the wait and its cutoff. ## Verification - Red before the fix: the new test "waits for a writer that just ended while its save or sandbox release settles" failed. The run had `status: failed`, `errorCode: setup_failed`, `executionRecovery.kind: bootstrap`. This is the same result as the staging failures. - Green after the fix: `cd server && npx vitest run src/__tests__/heartbeat-remote-workspace-busy.test.ts src/services/remote-workspace-reservation.test.ts` (16 tests pass.) - Safety cases in the new tests: - The waiting run does not change the workspace authority. The unsaved writer's files and hold stay unchanged. - A writer that ended more than one hour ago still causes the unfinished-writer setup failure, with no scheduled retry. The wait is bounded. - Existing cases still pass: live writer wait with Stop and reassignment, the concurrent reservation race (`remote_workspace_writer_pending`), and the unused-reservation takeover. - Also run: `remote-workspaces.integration.test.ts`, `heartbeat-workspace-busy.test.ts`, `remote-workspace-save-concurrency.test.ts`, `workspace-recovery.test.ts`. All 6 suites pass (442 tests). - Not done: a live run on staging at this revision. The fix is not deployed. ## Risks - A task whose ended writer never settles now makes up to about one hour of short `workspace_busy` waits (one scheduled retry every 1 to 2 minutes) before it fails. Before, it failed at once. The end state after the cutoff is the same. - Some of the staging failures have other causes that this change does not fix: - A previous run that stays `running` with no output for more than one hour still causes this failure, as intended. - A dirty workspace whose writer ended without a saved result (for example a workspace finalization retry that never runs) still blocks the task after the cutoff. These need separate fixes. - No schema, API or UI changes. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, browser reads of staging run records) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-04 17:39:26 -06:00
lyuboslavmedarov lyuboslavmedarov Claude Opus 5.5	3634c4a845	fix(chat): log the underlying error when a publication becomes delivery_unknown (#18875 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Chat connectors (Slack and others) publish agent messages to external conversations through the publication worker > - When the provider accepts a message but the local commit after the send fails, the publication becomes `delivery_unknown` > - A `delivery_unknown` row blocks the later publications in the same conversation until an operator resolves it > - The log line for this case contains only the generic reason. It drops the error that the post-send commit threw > - Operators cannot see why the commit failed, so they cannot tell a race from a lease loss or another cause > - This pull request adds the underlying error, redacted and length-limited, to the existing warning for this case only > - The benefit is that operators and maintainers can find the real cause of stalled chat threads ## Linked Issues or Issue Description Refs #18874 I searched open and closed issues and pull requests for `delivery_unknown`, Slack publication failures and "could not confirm its durable result". I found no duplicate. #18874 describes the bug and the proposed behavior fix. This pull request does not change behavior. It only makes the cause visible. ## What Changed - `server/src/services/chat-channels/publications/failures.ts`: when `providerAccepted` is true, the `chat publication failed` warning gets a `cause` field. The field contains the underlying error message, redacted with `redactSensitiveText` and limited to `MAX_ERROR_TEXT`, like the existing `error` field. - The stored `redactedError`, the disposition and the operator resolution flow do not change. - `server/src/__tests__/chat-run-milestone-work.test.ts`: the existing `rechecks resolved prompt ownership after delivery ...` test now spies on `logger.warn`. It asserts that `delivery_unknown` logs the `cause`, and that `retry` and `failed` do not log it. ## Verification - Ran in a `node:24-bookworm-slim` container on this branch: - `npx vitest run src/__tests__/chat-run-milestone-work.test.ts` (in `server/`): 72 passed. - `pnpm run typecheck` (in `server/`): passed. - Negative check: I removed only the new `cause` field and ran the test again. The two `delivery_unknown` cases failed. The `retry` and `failed` cases passed. ## Risks - Low risk. The change adds one field to one warning log line, only for publications that the provider already accepted. - The new field uses the same redaction and length limit as the existing `error` field, so it does not add a new path for secrets into the log. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), 1M context window, in Claude Code with tool use (shell, file edits, code search). A human reviewed and requested the change. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: lyuboslavmedarov <285605601+lyuboslavmedarov@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-04 17:38:39 -06:00
Nicky LeachandClaude Opus 5.5	2c39a2866b	feat(plugins): let managed config grant worker variables to a plugin (#19049 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A managed instance gets its plugins from the control plane, and people do not install them > - Some of these plugins need deployment values, such as a service origin and a capability > - The plugin worker environment is scrubbed, and only a fixed list of first-party providers gets deployment variables > - Core also sends lifecycle events and project instance secrets only to plugins with a saved config row > - So a provisioned plugin cannot work until a person saves configuration for it > - This pull request lets the managed config grant named variables to a selected plugin > - The benefit is that a distribution plugin can run with no tenant configuration, and Core does not need to name it ## Linked Issues or Issue Description No issue exists. This describes the problem. I searched open pull requests for related work and found no duplicate. What existing behavior does this improve? How a plugin provisioned through `plugins.autoInstall` receives its deployment settings. Current behavior `buildPluginWorkerEnv` forwards variables only from a fixed list that names first-party packages. A distribution plugin cannot be on that list without Core naming it. A plugin with no `pluginConfig` row gets no resource lifecycle events, and project instance secrets refuse it. Proposed behavior The managed config can include `plugins.workerEnv`. It maps a catalog key to variable names. The worker of that plugin receives those variables from the server process environment. The managed config can also include `plugins.enabledCompanies`. It maps a catalog key to company IDs. In those companies, the plugin gets lifecycle events and can use project instance secrets with no config row. Reason and benefit The control plane stays the only place that decides which plugin gets which values. Tenants do not handle these values, and they are not in the database or in the managed config document. Alternatives considered - Create a config or enablement row for each company. The control plane would need a new tenant write API, and a row can stay after the document removes access or return after a database restore. The document is never saved, like the other managed settings. - Give a grant access to every company. This has no per-company opt-in. - Put the values in the managed config document. The document does not hold secrets, and this rule stays true because it holds only names. ## What Changed - `parseManagedConfigEnv` accepts an optional `plugins.workerEnv` object. Each key must be in `plugins.autoInstall`. Names must be uppercase variable names, with no duplicates. - Startup fails if a name looks like a host credential, such as a name with `KEY`, `TOKEN`, `SECRET`, `PASSWORD`, `DATABASE` or `JWT` as a word, or if it controls how the worker loads code, such as `NODE_OPTIONS`. - `parseManagedConfigEnv` accepts an optional `plugins.enabledCompanies` object. Each key must be in `plugins.autoInstall`. Values must be lowercase company UUIDs, with no duplicates. - `managedPluginWorkerEnvGrant` and `managedPluginEnabledCompanies` return values by plugin key. A managed instance only starts a selected plugin from the current image, so the saved package path is not part of this check. It can be old or null. - `buildPluginWorkerEnv` forwards granted variables. This does not depend on the `environment.drivers.register` capability. - Plugin lifecycle delivery includes a plugin with no config row only for the companies the document names. It still skips a company that disabled the plugin. The runtime event listener follows the same rule. - Project instance secrets accept a plugin with no config row only for a company the document names. - A variable grant alone gives no company access. - `doc/plugins/DISTRIBUTION-PLUGINS.md` describes both sections. ## Verification - `npx vitest run` in `server/` on `managed-config`, `managed-plugin-worker-env`, `plugin-lifecycle-delivery`, `project-instance-secrets`, `plugin-database` and `bundled-plugins`: 201 tests pass. - The new tests cover a digest-verified distribution plugin, a stale saved package path, a catalog key named after an inherited property, a company the document does not name, a company that disabled the plugin, a named company that does not exist, a grant with no enabled company, and host credential names. - `tsc --noEmit` in `server/` shows no errors in the changed files. My worktree does not have the built `@paperclipai/paperclip-runner` package, so unrelated files show errors locally. CI runs the full typecheck. - I did not run the full test suite locally. CI runs it. - I did not test this change on a running instance. ## Risks - Older builds: a build without this change refuses a document with `plugins.workerEnv` or `plugins.enabledCompanies`, because it fails closed on unknown keys. A control plane must send these sections only to a build that supports them. - Trust: the granted plugin gets the variable values in its process. The image owner and the control plane already trust this plugin. The name check stops a mistake with host credentials, but a name check cannot prove that a value is not sensitive. Core cannot use an allowlist, because it must not name the variables of a private plugin. - Company scope: the control plane sets which companies a plugin serves. A new company needs a new document, and the server reads the document when it starts. Self-hosted instances do not change. - Rotation: a new value reaches the worker only when the server restarts. Rotation without a restart is not part of this change. - Background calls: a grant does not add proactive company access. A plugin that makes background calls with no invocation still needs a saved config for that company. - This changes core plugin behavior, and I did not discuss it in `#dev` first. Maintainers can tell me if they want a discussion before review. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), through Claude Code, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-04 15:24:43 -08:00
Devin FoleyandPaperclip	c7390bef20	fix(server): stop one slow copy-back from starving workspace retries (#19111 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native run can accept its result before the workspace copy-back finishes. A background worker retries a failed copy-back from the retained sandbox without another provider turn. > - Until that retry succeeds, the run stays `running` and keeps its task's execution lock. User answers and wakes for that task stay queued. > - The worker checked one entry at a time (`batchSize: 1`, no concurrent checks). A slow sandbox export held every other task's due retry for the whole export. > - Work without a saved retry time used a deadline of `0`. Copy-backs that a restart interrupted always sorted ahead of retries that were due hours earlier. > - On a busy server that restarts often, scheduled retries never got to the front of the queue. Their tasks stayed stuck. > - This pull request runs up to four checks at once, one per task, and orders pending work by when it became pending. > - The benefit is that one slow export cannot stall unrelated tasks, and an old retry runs before newer work. ## Linked Issues or Issue Description Refs #17734 (added this worker), #18421 (the same concurrency pattern for native reply recovery) and #19063 (open; records why finalization failed a run). This pull request does not change their behavior. What happened? On a hosted instance, sandbox exports were slow for a few hours. A harness-state save timed out after about 5.5 minutes in 27 runs, and worker copy-back attempts took 2 to 34 minutes each. Eleven native runs then waited in `retryable_failure` (or `workspace_finalizing` with an expired lease) for more than three hours. Each one had exactly one copy-back attempt and no retry. The worker's operation receipts show its attempts ran strictly one after another. After each restart, the worker first took the copy-backs that the restart interrupted. They have no `nextAttemptAt`, so their deadline was `0`. The server restarted every 30 to 40 minutes, so the overdue retries never ran. One stuck run held a task whose user had answered a question card. The answer's wake stayed queued for the same three hours. Expected behavior A slow copy-back for one task does not delay due copy-backs for other tasks. Due work runs oldest first. New work without a saved retry time does not go ahead of older due retries. Steps to reproduce 1. Create two native runs with accepted results on different tasks. Put both in `retryable_failure` with an overdue `nextAttemptAt`. The first is older. 2. Make the copy-back for the older run block. 3. Start the worker. On `master` the second task's copy-back never runs while the first one blocks. The new worker test runs these steps against PostgreSQL. Paperclip version or commit `master` at `69dea878c0b`. Deployment mode Any server with native runs that use remote sandbox workspaces. ## What Changed - `server/src/services/native-runtime/native-workspace-finalization-work.ts`: - Remove `batchSize: 1`. Add `concurrentChecks: { limit: 4 }` keyed by `issue:<company>:<issue>`. Runs that share a task keep their copy-backs serialized. Ownership is still checked per run by the existing advisory lock and owner receipt. - Work without a saved retry time uses the coordinator's `updatedAt` as its deadline, not `0`. The value is clamped to now, because the database clock can be slightly ahead of the process clock. Saved retry times and lease expiry behave as before. - `doc/native-workspace-finalization-recovery.md`: one paragraph on worker concurrency and ordering. ## Verification - Failing first: "retries another task while an older copy-back is still running". On `master` the second task's copy-back never ran while the older one blocked (the test timed out after 15 s). Now the second task finalizes while the first is still blocked. After release, the first one also finalizes. - Failing first: "orders work without a saved retry behind older overdue retries". On `master` the interrupted copy-back had deadline `0` and sorted first. Now it sorts after the older overdue retry, and its deadline is not in the future. - `cd server && npx vitest run src/__tests__/native-workspace-finalization-work.test.ts src/__tests__/scoped-deadline-work.test.ts src/__tests__/server-startup-feedback-export.test.ts`: 3 files, 159 passed. After rebasing onto `69dea878c0b`, the first two files: 38 passed. - `cd server && npx tsc --noEmit -p tsconfig.json`: pass. - Not verified: the worker on a live instance under slow exports. The diagnosis comes from that instance's operation receipts and run events. ## Risks - Up to four copy-backs can now run at once on one server. Each one uses its own dedicated database connection and sandbox transfer. Live runs already finalize in parallel, so this is within the existing load pattern. - The batch size is now the default (25). A pass ends when its slowest check ends, but the other lanes keep processing the rest of the batch. - `updatedAt` changes on any coordinator write. A pending entry with new writes moves later in the order, but it stays due. - No schema migration. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) through Claude Code, with tool use (shell, file edits, test runs, read-only database and log queries). The runtime does not expose the context window or reasoning configuration. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-04 15:14:39 -08:00
Dotta Paperclip Devin Foley	801d9548e3	test(server): cover competing recovery action inserts (#18773 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Recovery owners record one active action for each task source. > - Two owners can read an empty source before either inserts its action. > - PostgreSQL rejects one insert, and Drizzle wraps that conflict. > - The production fix for wrapped conflicts already landed in #18885. > - This pull request adds a real database race test for that fix. ## Linked Issues or Issue Description Refs #18885 and #18544. Related recovery work: #18402 and #18705. What happened? [CI run 41585908789](https://github.com/paperclipai/paperclip/actions/runs/41585908789/job/125752385968) failed with `issue_recovery_actions_active_source_uq`. The recovery worker did not handle the wrapped PostgreSQL conflict. #18885 now provides the production fix. Expected behavior Both competing recovery owners return the same active action. The existing bounded retry handles the exact wrapped unique constraint. Steps to reproduce 1. Create a task in real PostgreSQL. 2. Hold both recovery inserts until both owners read an empty source. 3. Release both writes. 4. Check that the losing transaction records a wrapped `23505` conflict and both service calls return one action. ## What Changed - Retain the newer bounded cause-chain handling from #18885. - Add a deterministic database race test. Check the actual wrapped conflict, both returned action IDs, the saved action count, and the attempt count. - Keep the original real-clock settlement and idle-window assertions. ## Verification - The two owning suites passed 98/98 tests after resolving the conflict: `pnpm exec vitest run server/src/__tests__/issue-recovery-dependencies.test.ts server/src/__tests__/issue-recovery-actions.test.ts`. - Workspace typechecks passed across the root run and a successful server rerun. The first server attempt could not resolve the local plugin SDK from provider sources. Restoring that checkout-only SDK link fixed resolution. The server typecheck rerun passed. - Full local `pnpm test:run` has not run. The current local full build stopped after the host exhausted disk space during native compilation. Generated caches were removed from this checkout; no source or test setting changed. Fresh CI is complete: 53 successful checks, two expected skips and one neutral security scan. The branch is mergeable, and the exact-head Greptile review is 5/5 with zero unresolved threads. - Greptile scored current head `dbaee78272a5b0f1db87ba33340a63571ef16948` 5/5 with no actionable findings. - Historical red proof: before wrapped-conflict handling, this race returned one rejection. The losing transaction carried PostgreSQL code `23505` and constraint `issue_recovery_actions_active_source_uq`. - Preserve prior CI evidence: an [old serialized job](https://github.com/paperclipai/paperclip/actions/runs/41649267044/job/125964435813) hit an agent-audit setup timeout and a heartbeat wait failure. The exact timing cause remained unproven. Its [single rerun](https://github.com/paperclipai/paperclip/actions/runs/41649267044/job/125972675510) passed. This PR changes no waits or assertions in those tests. ## Risks - Low risk. The final diff adds one database regression test. It changes no production code, schema, API, permissions, runner routing, or workflows. ## Model Used - OpenAI GPT-6.1 Sol (`gpt-6.1-sol`), xhigh reasoning, tool use, and code execution. ## Checklist - [x] I have included a thinking path and specified the model used. - [x] I have searched for related work and linked the production fix. - [x] I have included only public issue and PR references. - [x] The changed test passes against real PostgreSQL. - [x] I have documented verification limits and risks. - [x] All Paperclip CI gates are green. - [x] Greptile is 5/5 with no unresolved comments on the final head. - [x] I will address review comments before requesting merge. Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Devin Foley <devin@paperclip.ing>	2026-11-04 14:47:29 -08:00
Devin Foley Claude Opus 5.5 Paperclip	c19c4348c5	fix(server): record inspector provenance for dynamic import callback errors (#17844 ) <!-- entire-trail-link-start --> https://entire.com/gh/paperclipai/-/trails/49/changes/paperclip/49 <!-- entire-trail-link-end --> ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The server sends errors to Sentry when an operator sets a backend DSN. An unhandled rejection is captured in `strict` mode, so the server process exits after the capture. > - A hosted server reported `TypeError [ERR_VM_DYNAMIC_IMPORT_CALLBACK_MISSING]: A dynamic import callback was not specified.` two times, eight days apart, on two unrelated releases. Each event stopped the process. > - The stack has no first-party frame. An anonymous `setTimeout` callback at `<anonymous>:1:N` calls `import()`. The column changed between the two events, so the source text changed. > - In local reproduction, only two paths make that frame: `vm.compileFunction` without a filename, and an expression that a client evaluates through the Node inspector. No shipped server code or production dependency uses either path. The current events give no data about the source. > - This pull request adds bounded provenance to that one error: the inspector state when the event is reported (never its URL), allow-listed Node flag names, the process ids, the main-thread flag and the uptime. > - The benefit is that the next event gives evidence about the source of the anonymous code. A `closed` state rules out an expression from a still-open inspector session. An `open` state needs other evidence before a debugger is blamed. ## Linked Issues or Issue Description No public issue exists. This section follows the bug report template. What happened? A hosted server process stopped two times with an unhandled rejection: `TypeError [ERR_VM_DYNAMIC_IMPORT_CALLBACK_MISSING]: A dynamic import callback was not specified.` The stack is `importModuleDynamicallyCallback (node:internal/modules/esm/utils:280)` ← `Timeout._onTimeout (<anonymous>:1:20)` (first event) or `(<anonymous>:1:41)` (second event) ← `listOnTimeout`. The events came 3 minutes and 17 minutes after process start, on two releases that changed only skill documents and UI files. The Sentry event has no data that shows where the anonymous code came from. Expected behavior The error report gives enough data to investigate the source of the anonymous code: whether an inspector session was open in the process, and which inspector-related Node flags were set. Steps to reproduce 1. `node -e "require('node:vm').compileFunction(\"setTimeout(() => import('node:fs'), 1)\", [])()"` gives the same error and the same `Timeout._onTimeout (<anonymous>:1:N)` frame. 2. An expression such as `setTimeout(() => import('node:fs'), 1)`, evaluated through the inspector (`Runtime.evaluate`), gives the same error and frame. 3. `vm.Script` (`evalmachine.<anonymous>`), `eval` or `new Function` in a `vm` context (`eval at …`), `new Function`, indirect `eval`, `node -e`, `node -p` and stdin give a different frame or import successfully. Paperclip version or commit `master` at `0cb3fc7db591ba23e21fbbc70042bc44ec7d6f20`. The hosted events came from builds `f96a1e64` and `3e7c3ee9`, on Node v24.22.0 and v24.22.1. Relevant subsystem Server Sentry error monitoring (`server/src/sentry.ts`). ## What Changed - New `server/src/sentry-dynamic-import-provenance.ts`: `addDynamicImportProvenance(event, hint)` acts only when `hint.originalException.code` is `ERR_VM_DYNAMIC_IMPORT_CALLBACK_MISSING`. It adds: - the tag `dynamic_import_inspector` with the value `open`, `closed` or `unavailable` - the context `dynamic_import_provenance` with `inspector`, `nodeFlags`, `mainThread`, `pid`, `ppid` and `uptimeSeconds` - `nodeFlags` holds only the names of `--inspect`, `--inspect-brk`, `--inspect-wait`, `--inspect-port`, `--debug-port`, `--disable-sigusr1` and `--experimental-vm-modules` from `execArgv` and `NODE_OPTIONS`. Flag values and all other flags are dropped. - The hook never sends the inspector URL, because the URL contains a session secret. It loads `node:inspector` lazily, so a Node build without the inspector still starts and reports `unavailable`. The hook never throws. Every other event is unchanged. - The module comment and the doc say that the state is evidence at report time, not proof of origin. - `server/src/sentry.ts`: `beforeSend` is now `enrichServerEvent`. It runs the existing run-link enrichment, then the new hook. - `doc/observability.md`: the default capture set names the new tag and context, and says what is never sent. - `.github/workflows/sentry-contract.yml`: the Real Sentry SDK isolation job also runs the new test file. - Tests in `server/src/__tests__/sentry-dynamic-import-provenance.test.ts`: - the real Node error from `vm.compileFunction` - a real open inspector session: the URL and its secret are absent from the event. The test closes only the session it opened, and it skips when a developer already has an inspector open. - flag filtering with values and other flags present in `NODE_OPTIONS` - other codes and other hint shapes stay unchanged - a failed state read keeps the original event - the real `@sentry/node@10.71.0` SDK sends the original rejection to the server `beforeSend`, and enriches only the matching event. With `PAPERCLIP_REQUIRE_SENTRY_TEST_SDK=1`, a missing SDK fails the file. ## Verification - With the audited SDK on `NODE_PATH` and `PAPERCLIP_REQUIRE_SENTRY_TEST_SDK=1`, as in the contract job: `run-failure-sentry-real-sdk.test.ts` and the new test file pass (17 tests). - Without the SDK: the new test file, `sentry.test.ts` and `sentry-links.test.ts` pass (52 pass, 6 skipped). - With `PAPERCLIP_REQUIRE_SENTRY_TEST_SDK=1` and no SDK: the new test file fails at once, as intended. - Red: with `beforeSend` set back to `addSentryRunLinks`, the real-SDK test fails (`expected [] to have a length of 1`). Green with the change. - Hosted CI on this head: typecheck, build, all test partitions and Real Sentry SDK isolation pass. - Not run locally: the full Vitest suite and browser suites. A live hosted check is not possible before deployment. ## Risks - Low risk. The hook acts only on one error code. All other events keep their current content and grouping. - The process still exits after the capture, as before. This change adds diagnostics. It does not fix the source of the anonymous code. - `pid`, `ppid` and uptime describe the server process only. They hold no tenant, user or request data. - If a later Node version changes the error code, the hook stops acting and the events are the same as today. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through the Claude Agent SDK, with tool use (shell, file edit, code execution, GitHub and Sentry connectors). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-04 14:07:26 -08:00
Devin Foley devinfoley Paperclip Bender Claude Fable 5.1	a3313f6944	fix(ui): complete read recovery and enforce query error rendering (#18822 ) ## Thinking Path > - Paperclip lets people manage agent work through the board. > - The board reads task, agent, settings, plugin, and run data from the server. > - A deploy can interrupt those reads after content is already visible. > - Local error branches can replace that content with an error or show raw codes. > - This pull request moves the remaining reads to the shared recovery behavior. > - The query-error guard prevents new local error patterns from returning. ## Linked Issues or Issue Description What happened? Loaded lists, settings, task panels, and manual reads could clear their content or show errors during a server restart. Some reads also disabled the shared retry policy. Expected behavior Keep loaded content during a temporary failure. Show the shared connection banner. Retry reads after recovery. Replace content after a real access denial or missing resource, with readable copy. Steps to reproduce Open a task, settings page, or run log. Return 503 with `tenant_app_unavailable` from the API. Restore the API and check that content refreshes. Repeat with 403 and 404 to check the failure state. Related merged work: Refs #15754, Refs #15772, Refs #17960. This change covers remaining reads and enforcement. Browser persistence and the outage browser suite follow separately. ## What Changed - Repair an existing PostgreSQL deadline fixture at UTC boundaries. A daily cap defers a heartbeat until both its original interval and the next UTC day allow it. Keep the counted run in its seeded day and cover noon, just before midnight and the first second after midnight with explicit clocks and real PostgreSQL. Production scheduling is unchanged. - Integrate the saved list, settings, shell, task, and manual-read recovery changes against current master. Preserve the current task, Inbox, and run-log structure. - Add `useResilientFetch` for component-local reads. Preserve loaded data, cancel stale requests, and retry through the shared connectivity policy. Keep browser and pairing capabilities in memory. - Retain run log chunks across stream changes and catch up after recovery without duplicate chunks or stalled requests. - Keep genuine authorization and missing-resource failures visible. Translate known single-word codes such as `forbidden`. Show known skills and disconnected-app failures before sibling loading states. Keep Retry bound to its failed request. - Make the query-error guard enforce by default and run it in the CI policy job. Document the narrow exceptions and the read recovery contract. - Invalidate pending OAuth discovery when the operator leaves setup or its context changes. Keep reserved-popup cleanup bound to the current attempt. ## Verification - Current read head `09d1db29dea` is merge-ready: all 59 check runs are completed (54 successful, three skipped, two neutral), the legacy Snyk status is successful, and both required `ci / verify` and `ci / e2e` gates pass. Greptile Apex reviewed this exact head at 5/5 with no unresolved threads; the branch has no merge conflicts. The current CI run is `41748339602`. - Current read head `09d1db29dea` includes master `683fa4725e9`. The latest change touches only the existing server deadline test fixture. Application source remains the previously verified `86376fa751e` revision: all 60 affected skills/disconnected-app tests and both application UI typechecks passed, along with token gates and the enforcing query-error guard. Those UI suites were not repeated for this test-only repair. - The fixture repair passes all seven deadline integration cases and the complete server typecheck, including the server test layer. The old assertion deterministically fails at 23:59:06.888 UTC; its one-second-old run also fails the first-second-of-day probe. The corrected fixture passes noon and both midnight probes. The permanent three-case coverage uses seeded 2099 clocks. - Twelve new regressions fail against the previous implementation. Both skills variants and disconnected-app pages show either request's 403 or 429 while the sibling pauses or remains fetching. Permission failures stay visible. Retry stays usable for its failed request and recovers independently. An initial test-fixture query-key typo was corrected before the passing run. The finding is replied to and resolved. Greptile reviewed prior application revision `86376fa751e` at 5/5 with no new actionable findings or unresolved threads. The current test-only head has now received its own Apex 5/5 review with no unresolved threads, and its CI is successful as recorded above. - The preceding OAuth fix passed all 237 setup tests on both application branches. Its ten regressions prove that Back, Cancel, unmount, or context changes retire initial discovery without creating a connection, caching old identities, navigating an abandoned popup, or closing a newer attempt. - Earlier master integration `e992f770339` passed all 309 tests across eight affected Slack, recovery, retry, submission and task files, plus UI typecheck, token gates, enforcing query-error scan and seven guard tests. The integration preserves Slack workspace reconnect and shared recovery, readable execution errors, source-inspection company scoping, and current submission behavior. Newly inherited retained-file recovery controls keep cached data through transient failures and hide it after definitive denial. - The preceding cache head `97c0bc3966f` passed 1,038 focused cache/write/editor/account/task/API tests. All 22 built-UI outage/write/loading cases passed in 4.7 minutes on exact browser head `b7deaaee5da`, with strict checking of all three specs. The latest skills/app rendering fix is covered by its affected component suites; those broader cases were not repeated for this separate read path. - Historical broader UI, CLI and build results and unresolved aggregate server/DB/adapter failures remain in the PR discussion. Interrupted CI runs were audited and recorded separately; they are not counted as passes. Controlled local outages do not prove hosted authentication or a real Cloud restart/deploy. Real hosted acceptance remains required after merge and deployment. Commands for the latest fix: ```sh pnpm --filter @paperclipai/ui exec vitest run src/pages/CompanySkills.test.tsx src/pages/apps/AppNotConnected.test.tsx pnpm --filter @paperclipai/ui typecheck pnpm check:token-gates node scripts/check-query-error-rendering.mjs --enforce --summary ``` ## Risks This migration touches many read surfaces. The main risks are initial loading, request cancellation, retained authorization state, and run-log paging. The tests cover those states. There are no server, schema, dependency, or API contract changes. The follow-up browser suite and hosted deploy acceptance are required before release acceptance. Recovered commits retain their original authors. If squash-merging, preserve `Co-Authored-By: Bender (Fable) <bender@agents.paperclip.ing>` and the Paperclip trailer. ## Model Used OpenAI `gpt-6.1-sol` through Codex, with xhigh reasoning effort, tool use, and code execution. The context window size is not exposed. Saved source commits keep their original authorship. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in-PR following the bug report template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal ticket id - [x] The affected local tests pass; broader limitations are recorded above - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com> Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Bender (Fable) <bender@agents.paperclip.ing> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>	2026-11-04 14:07:18 -08:00
DottaandPaperclip	52b4b3ccd7	test(evals): pin native credential delivery and worker receipt fixes (#19042 ) ## Thinking Path > - Paperclip must deliver agent work through ordinary human controls. > - The latest three-case run retained three failed originals. > - The personal value was saved, but the native OpenCode child did not receive the configured task environment. > - A worker proof also relied on legacy process identity fields outside that environment contract. > - Timeline and credential checks had independent review defects that could reject truthful outcomes or miss unsafe instructions. > - This PR pins the reviewed fixes together and keeps all prior grades unchanged. > - A new AWS measurement must still prove the real outcome. ## Linked Issues or Issue Description Refs #18963, #18967, #18968, #18971, #18972, #18977, #19027, #19031. Follows merged #18995. This is the next infrastructure pin, not another core implementation PR. The latest launch 2657 retained FAILED originals for timeline, approved-tool replay and personal secret. The personal journey reached secure UI save and a worker wake, but the worker reported no credential and made no endpoint call. These are failed attempts, not new passes. Six of the selected ten cases have independent passes across prior attempts. The uniform final ten and the full catalog remain unqualified. No old result was regraded. ## What Changed - Pin immutable candidate `c6cc59282a45ed3c79880a220f7f60aa3ac1d6a4` on `codex/chat-credential-destination-contract-candidate-20261104`. - Retain the frozen candidate baseline. Add the accepted failed-action completion feedback hint, canonical configured environment delivery to OpenCode, actual native worker output receipt checks, timeline attribution controls, and credential clause safety checks. The feedback change leaves the existing acceptance gate in place. - Declare exactly seven oracle-version changes: timeline v5 to v6; personal v5 to v6; company v4 to v5; secure API key, least-scope delivery, secret proposal and vault reference each external v1 to v2. The four external cases have explicit overrides; the shared default is unchanged. - Sync the manifest rows, explicit changed-field records, descriptor snapshot and expected target. Preserve all 244 identities, original queries, human/native counts, clocks, model, profile, launcher bytes, privacy limits and the existing 196/244 qualification gate. - Include the reviewed retry cancellation readiness fixture repair from `815d47e4b5d7756367d2c5e2da656b640273eac2` in this infrastructure PR only. It awaits the first scheduled batch before polling its state. Its production behavior and assertions are unchanged. This test repair is absent from the model candidate. - Keep the core PRs, including #19027 and #19031, unmerged. A normal merge of this pin does not merge those PRs. ## Verification - Pin-local checks: all 189 existing manifest, TestOps API, adapter, reconciliation and streaming controls passed. The reviewed 815 fixture repair passed its owning real-PG control (one selected test). A temporary delayed-first-batch control failed with the old observer and passed with the new observer. Batch limits, assertions, drain wait and overall budget are unchanged. The original CI cause remains unknown. - Final candidate: 89 credential/receipt consumer controls passed. The exact Runner TypeScript build, 68 driver controls and one owning failed-action PG control passed before the later helper-only changes. Their runtime inputs remain byte-identical; those results are reused explicitly, not claimed as new executions on every source commit. - Real pinned OpenCode 1.18.34 executed a canned local-provider bash call. Synthetic `PUBLISHING_API_KEY` reached its child in a 2.89-second passing control with zero paid provider calls. This proves server-to-bash inheritance; driver spawn controls cover the preceding projection boundary. Actual AWS secure-entry, worker delivery and privacy acceptance remain pending. - The latest local candidate declaration loader is NOT GREEN. It exceeded its unchanged 30-second limit in both a combined run and an isolated run. Five other declaration/contract checks passed. Both failures remain recorded. No timeout was widened and no further retry occurred. Descriptor and definition modules match the earlier passing 5aa source; only the transitive credential helper changed. Prior 242-declaration evidence supports unchanged definitions only, not a claim that these failed executions passed. - The snapshot matches the candidate and exactly seven oracle-version leaves differ from source 249. All 244 identities, queries, run/human counts, clocks, model, profile, launcher and privacy limits remain unchanged. The SDK guard reports 506 unchanged inputs, unchanged runtime pins and registry `efa70621e9fb42aa185f7ee616a1a726aa3eccb367e67e28817cedf26ef2ade8`. The driver is outside those SDK roots. No SDK renewal occurred. Reporting-lock and whitespace checks passed. - Genuine exact-head reviews: source 19e3 is 5/5 with no unresolved threads; current pin 988 is 5/5 with no unresolved threads. The prior pin 4/5 and resolved P1 remain historical. Current exact-head normal CI 41917215227 completed successfully: 48 successful jobs and one selector skip, including final verify job 126888275415. Windows 41917215440 succeeded. The full PR rollup is 56 successes and three intentional skips, with no pending or failed checks. Older green heads do not qualify this head. Paperclip has only an earlier-head 5/5 assessment, which is stale for this head and receives no current assessment credit. - Six of ten cases have independent passes across prior attempts. Launch 2657 retains three FAILED originals; no grade was changed. The next measurement is the exact uniform ten, cap eight and zero automatic retries. No uniform-ten or full-catalog acceptance is claimed. - The local-tests checkbox below covers only the 189 pin controls and owning fixture control. It does not claim all candidate or repository tests are green. Full local repository checks were not repeated for this infrastructure diff. Core PRs remain unmerged. No model call, eval dispatch, private archive read, fleet change or merge occurred for this update. ## Risks Environment delivery crosses a real provider process boundary, so secret isolation remains part of the next actual AWS acceptance. A worker receipt must match an actual successful native run and key-authenticated endpoint call; a copied boolean does not qualify. Oracle safety and attribution changes can alter acceptance, so balanced negatives remain required. No current uniform-ten pass is claimed. Normal CI and genuine exact-head review must finish before protected infrastructure merge. Root owns the next TestOps Submit, with the original model, limits, zero automatic retries and maximum eight concurrent case chains. ## Model Used OpenAI `gpt-6.1-sol` through Codex. Capabilities used: source analysis, mechanical integration, local code execution and PR preparation. The host does not expose the context-window limit. The evaluated model remains OpenCode/OpenRouter `deepseek/deepseek-v4.1-flash-1017`. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-04 15:52:12 -06:00
DottaandPaperclip	4740d251b7	fix(onboarding): retain plan approval across follow-up decisions (#19043 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Onboarding asks a person to approve a saved plan before execution tasks can start. > - A person can then approve a separate proposal for the first execution step. > - The creation gate checks only that latest card for the plan target. It loses the earlier approval of the unchanged plan. > - Decomposition and native context also omit checkbox plan confirmations. > - This pull request validates the latest execution decision and the latest plan decision together. > - The benefit is that approved work can start while human, revision, and proposal gates remain effective. ## Linked Issues or Issue Description Refs #17914, #18160, and #18283. The related-PR search found no duplicate repair. What happened? After a person accepted a checkbox plan card and a later execution confirmation, native task creation and REST child creation returned 403. Accepted-plan decomposition returned 422 because its lookup recognized only standard confirmations. Native task context returned no accepted plan for the checkbox decision. Expected behavior The latest execution proposal can use an independently approved, unchanged plan. A pending or rejected latest proposal must still stop execution. The current plan must retain valid human approval of its exact revision. Steps to reproduce 1. Save a plan on a server-created onboarding task. 2. Create a revision-bound checkbox confirmation and accept it as a human. 3. Create and accept a separate, untargeted confirmation for the first execution task. 4. Create the child through the native tool or REST API. 5. Separately, try accepted-plan decomposition or read native task context after step 2. Paperclip version or commit The workspace's registered hosted release is Core `09c05c55050fede448196637cbaa543a154da76e`. The three regressions also failed against local source `3cdb367369f29692b81cf19b514a2abdba90734f`. Deployment mode Hosted native Runner with ACPX Claude. Local regression proof uses actual PostgreSQL, authenticated HTTP routes, native tool authority, and synthetic task data on macOS arm64. No customer data or credentials are in the fixtures. ## What Changed - Validate the newest ordinary proposal and the newest plan proposal separately. Keep the existing human-card and conversational provenance checks for each. - Require the accepted plan target to match the current document and revision. A newer pending, rejected, invalid, or foreign plan decision cannot fall back to an older approval. - Share persisted confirmation parsing across admission, decomposition, and native context. Validate checkbox selections against the saved options and limits. - Return `acceptedPlan.approvedOptions` with the user's saved selected IDs and option text on later wakes, including revision-pinned context. Keep unchecked defaults out of scope. - Add behavioral and safety regressions. Document the approval rule. ## Verification - Three behavioral regressions failed before the repair: child creation returned 403, decomposition returned 422, and native context returned `acceptedPlan: null`. All three now pass. - Two scope-retention regressions failed before the review fix: native context omitted the selected option after a later approval, with and without a pinned plan revision. Both now pass. Unchecked defaults stay excluded. - Current admission, native-tool, and accepted-confirmation replay suites: 131 tests passed. Existing first-task completion safety checks: 17 passed, including the atomic decision recheck. - Safety cases cover missing and rejected plan approval, agent-only decisions, foreign source runs, changed revisions, invalid checkbox selections, newer pending or rejected proposals, and task-creation replay. - Full repository `pnpm -r typecheck` and `pnpm build` passed locally on final head `7c0e8ab784`. - [Full final-head CI](https://github.com/paperclipai/paperclip/actions/runs/41911580531) passed: all Vitest shards, Runner checks, eight browser E2E shards, build, release registry, and canary dry run. The required `verify` and `e2e` aggregates passed. Storybook checks were skipped for this backend change; the file-selection job was skipped because full CI ran. - Earlier local typecheck attempts exposed Copilot verification during build preparation, stale declarations across the rebase, and two test-fixture typing errors. The assets were verified, declarations rebuilt, and fixtures corrected before the complete final-head pass. - Full local Vitest attempts were interrupted for the rebase and review repair. The duplicate final local run was stopped after full Linux CI passed. No complete macOS Vitest-suite pass is claimed. - A fresh Claude-provider acceptance run remains unverified: the local Claude account is signed out and no Anthropic API credential is available. The actual native tool and HTTP paths are verified with real PostgreSQL persistence. This PR does not deploy or recover the reported production task. - Greptile: 5/5 on `7c0e8ab784`, with the review thread resolved. ## Risks A later acceptance must not revive rejected or changed scope. Admission validates both current decisions and retains source, company, human provenance, and document locks. The extra plan-proposal lookup is bounded and limited to accepted, untargeted first-task proposals. Ordinary delegation adds no new proof lookup. The native context adds an optional `approvedOptions` projection. Revision-pinned context now performs the existing scoped approval read to recover that selection. No database schema, request, provider, or background worker changes are included. ## Model Used OpenAI Codex, exact model `gpt-6.1-sol`, max reasoning effort, with repository, shell, browser, and GitHub tools. The current session metadata confirms the model and effort. The context window size is not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-04 15:24:58 -06:00
Nicky LeachandClaude Opus 5.5	be20d4ad97	fix(projects): keep projects on the storage chosen at creation (#19036 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A project stores its work in Core/local storage or in a storage plugin > - Today, Core finds the installed default each time it reads a project > - So installing a default storage plugin moves existing projects, without a saved change > - Agents that cannot reach the new storage then lose access to new workspaces > - This pull request makes Core apply the default once, when it creates a project > - The benefit is that a project stays on its storage until a person moves it ## Linked Issues or Issue Description No issue exists. This describes the problem. I searched open pull requests for related work and found no duplicate. What existing behavior does this improve? How Core chooses the storage for a project when a plugin claims `defaultForProjects`. Current behavior Installing such a plugin changes the storage of every existing project with no explicit choice. Disabling the plugin changes it back. Nothing is saved, so the project has no fixed owner. Execution environments and storage are separate contracts, so an agent can lose access to new workspaces on a project that it used before. Proposed behavior Core saves the storage choice on the project when it creates the project. A project with no saved choice uses Core/local storage. A plugin-backed project stays attached to its plugin. A Core/local project can move to a plugin. Reason and benefit A project stays on its storage until a person moves it, so installing or disabling a plugin does not change where work goes. Alternatives considered - Keep the default at read time and add a process that saves a choice on existing projects when a plugin arrives. This adds a background step and a race with new projects. - Add a new database column. The existing policy field already holds the choice, so a migration is not necessary. ## What Changed - `resolveProjectWorkspaceProvider` reads only the saved choice. It no longer applies an installed default. - `withDefaultProjectStorage` applies the default when Core creates a project, including a personal private project. With more than one default, a project with no explicit choice is refused. This is the existing conflict rule, now applied at creation. - `projectService.update` refuses a change of storage on a plugin-backed project. An update that omits the field keeps the attachment. - Plugin lifecycle delivery sends project events to a storage provider only for projects attached to it, and for deleted projects. Core acknowledges the other events for that provider. Other plugins continue to receive all project events. - The project settings control has no "Inherit" option. It is locked on a plugin-backed project. - `ProjectWorkspaceProviderConfiguration.source` no longer has the `"default"` value. - A project with no repository or folder ignores its saved storage choice when a task runs. A storage plugin cannot manage this type of project, so its tasks use a plain task directory. Without this change, the second run of a task reset the agent session, and remote agents could not use an empty task directory. - The plugin SDK README and the plugin authoring guide describe the new rules. ## Verification - `npx vitest run server/src/__tests__/project-workspace-provider.integration.test.ts server/src/__tests__/plugin-lifecycle-delivery.test.ts server/src/__tests__/plugin-workspaces.integration.test.ts packages/plugins/sdk/tests/workspace-provider.test.ts`: 93 tests pass. - `npx vitest run src/components/ProjectProperties.concurrency.test.tsx` in `ui/`: 9 tests pass. - `pnpm --filter @paperclipai/server typecheck` and `pnpm --filter @paperclipai/ui typecheck` pass. - `npx vitest run` on `heartbeat-session-preparation`, `workspace-preparation`, `remote-workspace-source`, `project-workspace-provider.integration`, and `plugin-workspaces.integration` in `server/`: 231 tests pass. - I did not run the full test suite locally. CI runs it. - I did not test this change on a running instance. ## Risks - Behavior change at upgrade: an existing project that uses a plugin only through the installed default, with no saved choice, moves to Core/local storage after upgrade. Its plugin files stay in place, but new work does not use them until an operator sets `storageProviderPluginId` on the project. This pull request has no automatic backfill. - Missed lifecycle events: a provider does not receive events for a Core/local project. If the project moves to that provider later, the update event starts its reconcile. - One-way move: a person cannot move a plugin-backed project back through the API or the UI. This is intended. If a plugin is disabled, the project fails closed, as it does today for an explicit choice. - This pull request does not stop a person from assigning an agent to a project with storage that the agent cannot reach. That is follow-up work. - This changes core plugin behavior, and I did not discuss it in `#dev` first. Maintainers can tell me if they want a discussion before review. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), through Claude Code, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-04 13:07:29 -08:00
Devin Foley devinfoley Claude Opus 5.5 Paperclip	4ea53f9eb5	fix(server): resolve issue identifiers in file-resources routes (#16352 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents and the board read task workspace files through the `/api/issues/:issueId/file-resources/` routes > - Clients can identify a task by its UUID or by its human identifier, for example `ABC-230` > - The issues router changes identifiers to UUIDs, but the file-resources router does not > - Thus an identifier goes into a `uuid` comparison, Postgres rejects the query, and the route returns 500 > - This pull request resolves identifiers in the file-resources router and returns 404 for malformed ids > - The benefit is that identifier links work, and bad ids give a correct 404 and no server error ## Linked Issues or Issue Description What happened?* A request to `GET /api/issues/ABC-230/file-resources/list` returned 500. The server log showed `invalid input syntax for type uuid` from `workspaceFileResourceService.getIssue`. Sentry recorded this error on hosted staging. Steps to reproduce 1. Create a task that has an identifier, for example `ABC-1`. 2. Send `GET /api/issues/ABC-1/file-resources/list?workspace=project`. 3. The server returns 500. The same request with the task UUID returns 200. Expected behavior The route accepts the identifier, as the other `/api/issues/:id` routes do. An unknown or malformed id returns 404. ## What Changed - Added `resolveIssueId` to `WorkspaceFileResourceService`. It changes a task identifier to the task UUID. It does not check visibility. `getIssue` still does the privacy-scoped read. - Added a `router.param("issueId")` handler to the file-resources router. It calls `resolveIssueId`, the same as the issues router does for its routes. - `getIssue` now returns 404 before the query when the id is not a UUID. This stops the Postgres cast error for all callers. - Added a regression test for identifier, lowercase identifier, private task, unknown identifier, and malformed id on the `content`, `list`, and `availability` routes. - Updated the test service fakes with the new method. ## Verification - `pnpm vitest run src/__tests__/file-resources.test.ts src/__tests__/file-resources-git-scan-load.test.ts` in `server/`: 55 passed. - Without the source change, the new test fails with `expected 200 "OK", got 500 "Internal Server Error"`. This reproduces the production error. - I did not run a full local `tsc` because the sandbox did not have sufficient memory. CI runs the typecheck. ## Risks - Low risk. The change adds one indexed lookup on `issues.identifier` (unique index) only when the path value has the identifier format. - The identifier lookup does not check visibility. `getIssue` then applies the privacy-scoped read, so a private or unknown task gives the same 404 as before. The route does not show if a task exists. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edit, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-04 12:30:26 -08:00
DottaandPaperclip	22735cfcea	fix(native): preserve governed terminal receipts during observer settlement (#19025 ) ## Thinking Path > - Paperclip manages work done by AI agents. > - A native turn can yield to a saved human question or approval. > - The provider must stop, and its terminal event must be saved. > - The old drain waits for both the event transaction and its later observers. > - A slow observer can make a saved terminal look unproved. > - This change separates the committed receipt from owned observer work. > - Checkpoint and result acceptance still wait for that work to settle. ## Linked Issues or Issue Description What happened? A native governed wait can fail with `governed_wait_provider_terminal_unproven` after its exact interrupted terminal is already in PostgreSQL. The event port waits for postcommit accounting, output and projection callbacks before it returns the receipt. Those callbacks can cross the provider drain deadline. What did you expect to happen? A committed terminal proves the event transaction. Its observers remain separate work. A checkpoint or result must not be accepted until those observers settle. A pending human-only question must keep its audience and continuation policy. Steps to reproduce Run the real PostgreSQL control-plane regression. Commit a same-turn interrupted terminal. Hold its postcommit observer beyond the existing provider drain. The parent code reports an unproved provider terminal. The fixed code retains the committed receipt and joins the observer before checkpoint or result acceptance. No overlapping repair was found in the current master or open-PR search. Related [#18977](https://github.com/paperclipai/paperclip/pull/18977) covers final conversation-response selection. It does not change this receipt boundary. The source reproduction is consistent with the observed failure. The exact hosted provider code is unavailable, so it does not prove that every observed failure used this branch. ## What Changed - Add an explicit terminal-only option to the internal event port. Ordinary appends keep their existing behavior. - Return that receipt only after the normal transaction and company, agent, run, session, source and revocation checks. - Retain the full observer promise. Join it before checkpoint and result writes. Keep its original rejection and observe it after an aborted join. - Reuse the existing provider drain and checkpoint deadlines. Preserve duplicate and restart replay without another model turn. - Document the receipt and duplicate-observer replay boundaries. Keep human-only and no-continuation cards unchanged. - Assert the actual Codex facade requests deferral only for its interrupted terminal. Isolate empty skill inventory in monitor-only process fixtures while retaining their real scheduler, lifecycle, selection snapshot and three-second deadline. ## Verification - Meaningful parent red: actual native runtime and real PostgreSQL event port. The terminal row commits, but a held observer causes `NativeProviderTerminalFailure` with `governed_wait_provider_terminal_unproven`. - Current candidate `0db24ae74ee42fdf92035d8407fc095e9f73cc6a`: full runtime suite, 203 passed. Real PostgreSQL control-plane, timing and monitor suites, 103 passed. Runner TypeScript and full layered server typechecks passed on the unchanged candidate source tree. The controls cover uncommitted and foreign evidence, explicit false and failed terminals, held and rejecting observers, abort and controller revocation, duplicate receipts, and new-port recovery. The real pending human-only card stays unanswered with continuation policy `none`. Recovery does not call `startTurn` again. - `pnpm -r typecheck`, the production server build (including the real Runner release binary), module boundaries and vendor dependency verification passed on `6d75`; the follow-up changes only tests and documentation. Diff checks pass on the current candidate. The prior `399ad3dc2b` repository-wide `test:run` was stopped as obsolete during the general-server phase after preserving its progress (exit 130, incomplete); its repository build was not reached. Its six completed chat shards passed 1,904 cases, but this is prior-head evidence and not a full-suite pass. Full current-head CI is required before merge. - CI attempt 1 on `6d75` failed one real three-second monitor-settlement assertion; the aggregate verify gate consequently failed. Other jobs passed. The actual CI merge tree was `3037bf6db162da3b0a9172046360401f8449ebc5` (`6d75` into `89703c3`). An unchanged full-file replay reproduced three timeouts. Original-deadline inspection found the legacy process still preparing; local phase timing measured cold skill inventory at 6.629 seconds. The original CI did not retain that phase detail, so its exact subcause remains unproved. The fixture correction supplies empty skill entries for these skill-free process tests and still invokes the selection-snapshot callback. It preserves all assertions and the three-second deadline, with no new drain wait. All 21 cases pass; deliberately restoring the old blocked-status rejection still fails the real blocked-check case with 409. No CI retry was issued. - The prior current-head native review was 4/5 with no production correctness finding. Its requested direct facade-options assertion and duplicate-observer wording correction are included. Current-head CI is green (55 success, 3 skipped), including required verify and E2E; current-head Greptile is 5/5 with no actionable findings. The old native advisory remains tied to `6d75`, and its requested findings are addressed. No new native assessment is claimed or required by protection. - The first draft broke three existing transaction-start timing controls. The no-debt path now keeps the original synchronous start, and all three pass. Two draft type errors were corrected. A further real-PG red found an extra default-append assertion after commit. Removing it restores the original committed receipt after a late abort; the same aborted signal still cannot write a checkpoint or result. Original setup and assertion failures remain in private verification receipts. - Normal merge of current master `3a2249dd5768313f04b2aa3860dfd363492eda16` has no conflicts. The affected production source blobs did not change in that baseline merge. - Local proof uses Node 24.22.2 on Darwin arm64, a real PostgreSQL instance and deterministic provider-boundary fixtures. It makes no model call. No deployed replay or original-task recovery is claimed. ## Risks Deferred observers must remain owned. A missed join could accept incomplete accounting or output. A rejected or stalled observer must prevent acceptance, and an abort must prevent late checkpoint/result writes. The tests exercise these boundaries. The port keeps the existing best-effort trace-publication handling. This change does not turn a PRP terminal into physical-stop or accounting-completion proof. ## Model Used OpenAI `gpt-6.1-sol`, with code and shell tools. The host does not expose the reasoning mode or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-04 14:11:03 -06:00
DottaandPaperclip	d180ad8c77	fix(heartbeat): retry chat execution authorization contention (#19017 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A task Retry must use the current task and chat authority. > - The request and queue paths already retain a Retry through brief lock contention. > - Execution checks that authority twice, including after native input preparation. > - A busy task row or chat identity could fail the admitted run at either check. > - This pull request retries only the rolled-back authorization transaction. > - The same run can proceed after the lock clears. The final dispatch gate still rejects changed ownership. ## Linked Issues or Issue Description What happened? A failed chat task can accept a normal Retry and then fail during execution setup when another PostgreSQL transaction holds the task row or chat identity lock. The late check can fail after native input and workspace preparation. The exception can then enter the workspace preservation path. Expected behavior Keep the same admitted run and its frozen input. Retry a recognized, rolled-back authorization conflict within the existing bound. Recheck current authority before each attempt. Reject Stop, paused agents, revoked chat access, or a changed task owner before the execution callback. Steps to reproduce 1. Admit a normal failed-chat task Retry with the existing PostgreSQL fixture. 2. On a second connection, hold the task row or the chat identity advisory lock when execution authorization starts. 3. Release the lock after the real authorizer reports contention. 4. Observe that the original code fails the run before the authorized callback. Repeat at the late native check after input preparation. Paperclip version or commit Reproduced on master `039e0dad2c51c80251e1302a19c5566e039c357a`. The branch is rebased on `8f06bd4e1`. Deployment mode Source checkout with real PostgreSQL and the local HTTP Retry route. Provider and chat transport responses use controlled test fixtures. Refs: #18895 and #18896 cover request and queue admission. This change covers execution authorization. #18898 and #18989 cover predecessor cleanup, a separate boundary. The roadmap and public PR search show no duplicate execution-authorization repair. ## What Changed - Recognize the existing typed chat-presentation contention error in the bounded admission retry helper. Keep its existing PostgreSQL lock-unavailable handling. - Use that helper at both execution authorization checks. Each attempt checks cancellation and the applicable legacy controller lease, then opens a fresh transaction. The loop contains no provider or workspace callback. - Include failed-chat Retry in the existing final dispatch ownership gate. This preserves current task ownership and native recovery owner checks after a delay. - Add real PostgreSQL contention, late prepared-input, Stop, pause, revocation, owner replacement, native completion, and ordinary legacy completion controls. Document the retry and dispatch boundaries. ## Verification - The unchanged base failed real task-row and chat-identity contention cases. A separate late native case failed after frozen input preparation. A safety control then proved that a replaced task owner could still reach the backend; the final dispatch gate fixes that path. - All 11 PostgreSQL cases and all four helper controls pass. The tests keep exact frozen model and trust input, reject changed authority, and do not replay a provider callback. - A real local HTTP Retry reaches the native executor, one controlled provider turn, one accepted result, a committed finalizer, and one published chat reply. Reprocessing the canonical request does not start another turn. - A separate real HTTP ordinary legacy Retry runs one controlled adapter callback, completes its task through the normal service, and does not replay the canonical request. - Independent review found no material issue in the two production files. It checked transaction scope, lock order, current ownership, native lease expiry, and import cycles. - The full workspace typecheck and full `pnpm build` pass on this head. Full `pnpm test:run` passed all 1,916 chat integration cases. Its broader server group reported the existing workspace-runtime case `returns a bounded structured conflict and exposes only same-company workspace references` as failed after 46 ms. The unchanged focused case and the complete 179-case file both pass on rerun. The active broad run was then stopped for this finite handoff; its child exited 130 and the command exited 1. The detailed assertion was buffered and was not retained, so its exact cause remains unknown. This is a partial local run, not a full local pass; later local groups did not run. All three CI chat shards pass. [Greptile reviewed this exact head at 5/5 with no findings](https://github.com/paperclipai/paperclip/pull/19017#issuecomment-6638521119). - Retained setup and fixture failures are not counted as product regressions or passes. An incomplete copied Copilot runtime failed the first typecheck; the supported installer repaired that isolated dependency, then typecheck passed. Early acceptance fixtures used a wrong publication field or did not complete their task; the final controls use the real field and completion service. - The first CI run failed [workspace shard 1](https://github.com/paperclipai/paperclip/actions/runs/41885475363/job/126769239992) on child-server startup and a five-second dense ZIP test deadline. [Server shard 2](https://github.com/paperclipai/paperclip/actions/runs/41885475363/job/126769239908) failed an existing local runtime readiness probe. All 18 cases in the two CLI files and the runtime case then passed unchanged locally on this head. The CLI files also match an earlier independently verified head byte for byte. One failed-job rerun was requested. Its [server job](https://github.com/paperclipai/paperclip/actions/runs/41885475363/job/126777070120) was cancelled after the runner received a shutdown signal. It produced no test verdict. After that workflow ended, only the interrupted server job and its required dependents were rescheduled once. The [complete workspace rerun passed](https://github.com/paperclipai/paperclip/actions/runs/41885475363/job/126777070125). The [rescheduled server job](https://github.com/paperclipai/paperclip/actions/runs/41885475363/job/126781960353) and its aggregate passed. The [complete current-head workflow is green](https://github.com/paperclipai/paperclip/actions/runs/41885475363). All required current-head CI and security gates pass, including the Ready-triggered scans; three conditional checks are skipped. The complete unchanged idle-history file also passes all 75 cases locally. No timeout or assertion changed; all attempts remain part of this result. - These local controls do not claim hosted-provider or Linux sandbox acceptance. The original task-row failure did not retain a SQLSTATE; the PostgreSQL reproduction proves the local gap without assigning that original cause. ## Risks - The existing bound allows 50 backoffs of 100 ms. It limits backoff time, not total database or lease wall time. - Native ownership is checked at the final dispatch gate. The per-attempt controller lease assertion applies to legacy execution. - A persistent lock or changed authority still fails closed. Unknown prior provider effects and accepted work retain their existing guards. - No migration or model, account, trust, credential, or permission policy change is included. ## Model Used OpenAI GPT-6 through Codex, with reasoning, tool use, code editing, and PostgreSQL verification. The host does not expose the exact serving model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass within the stated scope: relevant regression and acceptance controls, and the complete chat, idle-history, and workspace-runtime files. The broad local run remains partial as recorded above. - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-04 13:07:50 -06:00
Devin FoleyandPaperclip	edb9d29942	fix(recovery): preserve Retry authority and reopen cross-task cases (#18997 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Tasks keep execution identity and saved workspace files separate from control-plane updates. > - A scanner can have permission to reopen a case while its run cannot restore that case's files. > - Retry can also follow a preparation failure that has no recovery-action record. > - Both paths can accept valid work but leave the target unable to run. > - This change restores files under the target executor and saves admitted Retry authority on the canonical receipt. > - Existing workspace, user, source and successor checks still apply. ## Linked Issues or Issue Description What happened? An authorized agent updates another closed task. The route tries to restore the target workspace under the caller's run identity and returns `403 Task workspace run identity is unavailable`. A separate failure occurs when a normal board Retry follows a pre-provider preparation failure. Admission has no recovery-action row on which to save continuation authority. Dispatch then fails with `continuation_user_authorization_missing`. Expected behavior Accept the authorized task update. Restore its terminal workspace when the target's admitted executor starts. An admitted Retry must retain its exact dispatch authority even when the predecessor has no recovery-action record. Steps to reproduce 1. Complete a task with an isolated workspace. From a different running task, use an authorized comment or PATCH with `resume: true` on that closed task. 2. Separately, fail native preparation before provider work starts, with no recovery-action row. Submit board Retry, admit its successor, and build the execution continuation. 3. Before this change, the first path returns 403 and the second rejects its admitted authority. Paperclip version or commit Reproduced against `e968131efa7e63d321f3bcca8d47a62513c2efb4`. Rebased onto `afbc5a885b5`. Deployment mode Server-managed native task execution. Tests use real PostgreSQL and temporary workspace files. Related PRs: #16927 covers terminal workspace races. #17916 retains exact native export sources. Neither covers these route and Retry receipt failures. A GitHub search found no duplicate. ROADMAP.md was checked; this is a recovery bug fix. ## What Changed - Defer workspace restoration only for authorized cross-task comments and updates from a live run. Checkout keeps its existing workspace identity checks. - Restore the target's selected terminal workspace under its own admitted run. Preserve its saved files and existing source checks. Leave manually archived workspaces unchanged. - Store server-created continuation authority on the canonical admitted Retry receipt when no source recovery-action record exists. Existing action records remain authoritative. Read the fallback only after matching the company, task, agent, user, source, generation and successor. - Add route and PostgreSQL regression tests. Document both paths. ## Verification - Red: both cross-task route tests returned the observed 403 before the route fix. - Red: the target executor could not reuse its archived terminal workspace before the restoration fix. - Red: an admitted preparation Retry with no recovery-action row failed with `continuation_user_authorization_missing` before the receipt fix. - Green: 419 recovery and continuation tests pass, plus 183 focused route, environment and Retry tests. The assembled heartbeat test starts the agent in the same terminal worktree and verifies both its unpushed commit and uncommitted file bytes. It fails before the restoration fix. - Negative cases reject wrong Retry runs, users, source runs, generations, unadmitted receipts and missing authority. The original failed run stays unchanged. Workspace tests verify saved file bytes. - Full `pnpm build`, `pnpm -r typecheck`, `pnpm check:module-boundaries` and `git diff --check` pass. - The full suite found an existing assembled chat authority regression: clearing a recovery-action record could be bypassed by the new fallback. The fix now writes the fallback only when that source has no action record. That assembled regression passes with the original test unchanged. - Full `pnpm test:run` is running again. Its first attempt also hit an unrelated ancestor skill-root lookup for a missing Slack skill. The rerun uses a temporary untracked link to the repository's own bundled skills. This link is excluded from the PR. Full-suite completion and hosted CI are pending. - Hosted CI has multiple runner interruptions. E2e shard 1/8, server shards 3/12, 8/12 and 12/12, and workspaces-a shard 1/2 show runner shutdown signals. E2e shard 5/8 and server shards 1/12 and 2/12 have GitHub annotations reporting lost runner communication; their logs return 404. The `ci / verify` assertion failed because the general-test matrix was cancelled. GitHub rejected job and failed-workflow reruns while this workflow is still running. Rerun the interrupted checks after it settles. - Greptile reviewed exact head `a1e3929aedb1ee3bfcd001f4ebb1510106bf4070` at 2026-11-04 17:37 UTC: 5/5, with no new blocking findings and no unresolved review threads. ## Risks - A deferred workspace restore can still fail at target admission. The run keeps the existing workspace failure path; this change does not bypass file recovery or provider stop checks. - Retry receipts now store server-created authority. Existing action receipts remain valid. No schema migration is required. - Local regression tests establish the software behavior. The new revision has not been deployed to a hosted instance. ## Model Used OpenAI `gpt-6.1-sol` through the Codex harness in T3 Code, with xhigh reasoning, repository inspection, tool use and code execution. The context-window size is not exposed by the runtime. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-04 11:55:16 -06:00
DottaandPaperclip	b99dd57c82	test(evals): run Product regression from TestOps (#18983 ) ## Thinking Path > - Paperclip manages agent work through governed product workflows. > - Product E2E tests check those workflows through the real app and runner. > - The saved Product regression plan has 20 imported case identities. > - The plan has no live executor, so historical baseline rows do not qualify its current behavior. > - The existing paid full-stack workflow already owns all 20 required journeys. > - This change binds TestOps selection and reporting to that workflow without changing the product oracle or paid admission. > - Maintainers can run the plan and inspect current case verdicts and retained evidence. ## Linked Issues or Issue Description What happened? The standing Product regression plan has no executor. Its dashboard points at a historical 355-result snapshot across several suites and revisions. Expected behavior A TestOps plan run must select the intended existing full-stack cells, execute them on GitHub, and report their original case IDs, assertions and source revision. Missing evidence must remain a visible failure. Steps to reproduce Open the Product regression plan and inspect Executors. There is no live job. The existing full-stack workflow accepts exact cell IDs but has no TestOps selection or reporting path. Paperclip version or commit The branch includes reviewed master correction `04a24f4e06`. This change does not alter scenario prompts, models, deadlines or assertions. Related public work: #18542 establishes the separate chat-catalog lane. #18666 repairs its workflow URL binding. These changes do not implement this Product plan. ## What Changed - Add optional TestOps service inputs and a manual bootstrap selector to the existing full-stack workflow. - Map the original 20 case IDs and full names to existing native Codex/local cells, with the existing native OpenCode/local task-workspaces case. - Keep reporting tokens in trusted default-branch selection and publication jobs. Keep the existing paid provider authorization and protected environments. - Bind retained results to the workflow, job run, launch, source revision and selection hash. Preserve missing-result failures and earlier attempt verdict receipts. - Verify uploaded identities, coverage and verdicts. Document full-plan and subset reporting limits. - Keep the outstanding-decision reservation live while its existing deletion-fence test asserts the original 409 guard. Release the test provider in finally. Production deletion and accounting code stay unchanged. ## Verification - `node --test tests/runner-e2e/product-regression-testops.test.mjs`: 13 positive and negative controls pass, including the separated CLI handoff. - Existing paid-workflow security regression passes. - `actionlint -shellcheck= .github/workflows/runner-full-stack-e2e.yml`: passes. - The Product E2E catalog compatibility test confirms all 20 exact cells exist: passes. - `pnpm test:e2e:runner:typecheck`: passes. - Fresh Greptile review of final head `61ef7a6e` is 5/5 (11 files, no new findings); both initial reporting findings are repaired and resolved. - `server/src/__tests__/decision-models.test.ts`: all 37 controls pass on a disposable official PostgreSQL 18 container in 22.02 seconds. The unchanged focused case passed once and all five local samples; the original Linux CI failure remains retained. This is not a claimed deterministic local red reproduction. The new gate explicitly preserves the outstanding-reservation precondition. - The existing master readiness-fixture correction passes all three retained-workspace controls on the same database in 25.79 seconds. It is incorporated from master, not duplicated in this PR's diff. - Final-head Linux CI also passed all 37 decision-model controls in 7.047 seconds; the server shard passed 120 files and 2,004 tests. See [the retained successful shard](https://github.com/paperclipai/paperclip/actions/runs/41870214517/job/126715358759). - Previous CI failures are retained: [three stale readiness-fixture cases](https://github.com/paperclipai/paperclip/actions/runs/41864061890/job/126694812885) and [the reservation fixture race](https://github.com/paperclipai/paperclip/actions/runs/41864061890/job/126694812894). The separate [Canary Dry Run](https://github.com/paperclipai/paperclip/actions/runs/41864061890/job/126694811605) recorded a lost self-hosted runner connection before any step. The fixture repairs are verified on changed source; the separate Spot-interrupted CI attempt and its bounded retry are recorded below. - Actual one-cell GitHub canary on the unchanged product source: https://github.com/paperclipai/paperclip/actions/runs/41860189801 . The native Codex/local message-marker case passed all six assertions and cleanup in 35.24 seconds. The persisted issue is Done and its one run succeeded. This is supporting evidence; it is not TestOps acceptance. - Final-head CI attempt 1 was interrupted: three failed jobs have EC2 `Server.SpotInstanceTermination` receipts; six cancelled jobs show the same simultaneous shutdown. Original logs are retained. One [failed-job retry](https://github.com/paperclipai/paperclip/actions/runs/41870214517/attempts/2) passed all nine interrupted checks while preserving prior successful jobs. - Final-head full repository CI passes, including both required `ci / verify` and `ci / e2e` gates. GitHub still requires a non-author code-owner approval for the workflow change; normal auto-merge is queued. A fresh TestOps-triggered subset/full-plan acceptance remains pending. Local projection checks do not establish a live Product pass. ## Risks - The first bootstrap upload must establish the official TestOps job before the saved plan can use it. - This lane tests one declared local native configuration per case. It does not qualify the historical 355-row matrix. - Upload failures remain visible. A missing retained report produces broken case results and cannot pass. - The existing non-TestOps campaign path must keep its selection and scheduling behavior; paid-workflow regression coverage remains in place. ## Model Used OpenAI Codex with reasoning, code execution and tool use. The host does not expose the exact model ID or context window to this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-04 11:17:39 -06:00
DottaandPaperclip	6364fa7327	fix(heartbeat): wait for retry predecessor cleanup (#18989 ) ## Thinking Path > - Paperclip manages agent work and preserves its task ownership. > - Heartbeat admission controls when a queued run can start. > - A failed run can still own workspace saving and environment cleanup. > - Automatic recovery can transfer the task execution lock to its queued retry before that cleanup finishes. > - The old queue check then sees the retry itself and misses the predecessor's remaining work. > - This pull request checks the exact retry predecessor before claiming the queued run. > - Existing cleanup notifications can then resume that same run through normal admission. ## Linked Issues or Issue Description Refs #18898 and #18691. The first repair covers canonical connected-task retries with a native source. The second covers live remote workspace writers. This change covers ordinary automatic continuation when the predecessor failed before native runtime selection. The search found no open repair for that path. What happened? A preparation failure saved a terminal legacy run row while its remote workspace save and environment cleanup were still active. Generic continuation created one queued successor and moved the task execution lock to it. The queue checked that current lock but did not check the successor's predecessor. The new run started and failed at the existing unfinished-writer guard. Expected behavior Keep that same successor queued until its exact predecessor finishes execution and cleanup. Check company, agent and task identity before using retry provenance as a wait condition. Preserve Stop, reassignment, failed cleanup and independent recovery gates. Steps to reproduce 1. Save a failed legacy preparation run with an active environment lease. 2. Queue its ordinary automatic continuation and bind the task execution lock to the successor. 3. Claim the queued run before the predecessor finishes cleanup. 4. The old code starts the successor. The fixed code retains it in the queue. 5. Commit successful cleanup through the existing cleanup owner. The ordinary dispatcher starts that same run once. Paperclip version or commit The causal regression fails on master `86814c6550f52e99ede150a883bdc773cbb540e3`. Deployment mode Source checkout on macOS arm64 with disposable PostgreSQL, the assembled heartbeat service and the production work coordinator. The adapter boundary is controlled. No hosted provider replay or deployment is claimed. ## What Changed - Check the current task execution owner and the retry's exact predecessor under the existing task claim lock. - Use the saved native task binding, ordinary task binding and historical context fallback in that order. Ignore unrelated retry provenance. - Reuse the existing live-executor and pending-cleanup checks. Reuse committed cleanup notifications and the existing dispatcher. - Add real PostgreSQL claim controls and assembled cleanup-event acceptance. Keep the original source failure, saved files and current task ownership intact. - Document the queued predecessor-cleanup rule. ## Verification - RED: the unchanged queue claimed the successor and wrote a new controller lease while its legacy predecessor retained an active remote environment lease. The regression expected the successor to remain queued. - GREEN: the same regression passes after the change. The complete queue suite passes all 71 tests. - The assembled heartbeat suite passes all six tests. The added cases cover one continuation after committed cleanup, saved-file reuse, final task lock release, board Stop, reassignment and failed cleanup. - Scope controls cover a deleted source after scan, another company, another task, another agent, canonical native task binding and the existing current-owner hold. A failed cleanup and a still-live local executor retain the wait. - Node 24.19.0 and pnpm 9.15.4. Module boundaries and `git diff --check` pass. - Initial fixture attempts failed on database constraints and observer timing. The final fixtures use valid immutable bindings, the board Stop attribution and a completed real dispatcher call after reading the exact committed lease state. Production ownership guards were retained throughout. - Full low-memory repository typecheck passed at `5cddb49322`. The sync to master `04a24f4e06` adds only the independent retained-allocation test fixture from #18984; production code is unchanged. - Full `pnpm build` passes on current head `c7c2bcf08ffd05be5ce714683ae1dcfe6f5b49f1`. - The post-sync run executes all 80 behavior assertions successfully across the queue, assembled cleanup and updated upstream retention suites. Its queue `afterAll` then exceeds the existing 30-second database shutdown hook. That failed suite result is retained. The controlled post-build queue rerun passes all 71 tests with normal teardown. The earlier shutdown cause remains undetermined. Its owned queue-test PostgreSQL process has exited. The separate local default `pnpm test:run` was intentionally stopped after full unfiltered CI passed on the exact head. Four local chat partitions completed before that stop; this is not a local full-suite pass. The verified owned process group and its PostgreSQL fixture have exited. Logs and retained fixture data remain available. The timeout has not been increased. - Full unfiltered CI passes on current head `c7c2bcf08ffd05be5ce714683ae1dcfe6f5b49f1`: all 12 server shards, all six internal chat partitions, all nine serialized server shards, the workspace groups, native Runner lanes, typecheck, build and browser E2E. Both required [ci / verify](https://github.com/paperclipai/paperclip/actions/runs/41866478617/job/126711606613) and [ci / e2e](https://github.com/paperclipai/paperclip/actions/runs/41866478617/job/126711263146) pass. The file selector is skipped, so this is the full CI test matrix. The head has 55 successful checks and three expected skipped checks. The ready-for-review contributor trust and security scan also pass. - Current-head Greptile is 5/5, its check is successful, and no inline review threads remain. The ordinary review check also passes. No merge, deployment or original task retry was performed. ## Risks - A predecessor with unfinished cleanup keeps its successor queued. Its existing cleanup owner remains responsible for settlement. - A task claim reads at most one additional predecessor row. A matching source also uses the existing ownership and cleanup queries. The change adds no timer, schema, credential, provider or model policy. - A committed cleanup signal only requests normal admission. Current ownership and independent execution gates still apply. ## Model Used OpenAI GPT-6 through Codex, with reasoning, code editing, tool use and PostgreSQL verification. The session does not expose a more specific serving model ID or context-window limit. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass — focused suites pass; the separate local full-suite attempt is explicitly incomplete above - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-04 11:16:54 -06:00
DottaandPaperclip	04a24f4e06	test(recovery): align retained allocation fixture with readiness (#18984 ) ## Thinking Path > - Paperclip manages AI agents and their work. > - Restart recovery must preserve the original remote allocation and its files. > - #18934 readies that same retained allocation before preparation. > - The older #18936 failure fixture rejects every readiness call. > - It stops before the intended preparation-failure boundary. > - This change updates only that fixture and checks the original allocation identity. ## Linked Issues or Issue Description What happened? The three remote preparation retention cases fail after #18934. Their driver stub throws `Reattachment must not resume the allocation`. The production code now calls readiness on the same validated lease before preparation, so the test never reaches its intended execution failure. Expected behavior The fixture should allow readiness of the original allocation. It must reject replacement acquisition and retain its preparation-error, ownership, saved-file and retention checks. Steps to reproduce Run `src/__tests__/heartbeat-native-remote-failure-retention.test.ts` on master `86814c6550`. All three cases stop at the stale readiness stub. Refs #18934 and #18936. This one-file baseline correction is split from #18970 so it can unblock the shared failure also observed on #18977. It carries no core API change or evaluation source pin. ## What Changed - Return the original lease from the fixture readiness stub. - Assert exactly one readiness call for the original company, environment, run, workspace, provider allocation and remote path. - Assert that no new allocation is acquired. Keep all existing preparation failure, ownership and retained-work assertions. ## Verification - Before #18934, the three existing cases passed locally. On the actual merged #18934 contract, they reproduced the three CI failures in 18.06 seconds. - With this fixture correction, all three passed in 11.83 seconds. Production runtime code and the original 60-second case deadline are unchanged. - The corrected test blob and its relevant production/helper blobs match the already-tested merged-source proof. Current master still has the same stale test blob; this isolated PR applies only the identical fixture correction. - Command from `server/`: `node ../node_modules/vitest/vitest.mjs run --config vitest.config.ts src/__tests__/heartbeat-native-remote-failure-retention.test.ts`. - `git diff --check` passes. Full repository typecheck/build were not rerun for this one-file fixture split; fresh normal CI and review are pending. - [Original failing CI job](https://github.com/paperclipai/paperclip/actions/runs/41855440307/job/126665077718). It remains failed; no remote retry was used as proof. ## Risks - Low scope: test fixture only. A readiness call for a replacement or wrong allocation fails the identity assertions. - This does not alter production recovery, permissions, cleanup policy, evaluation deadlines or grading. ## Model Used OpenAI `gpt-6.1-sol`, with reasoning, repository inspection, code editing and tool execution. The host did not expose the model context-window limit. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-04 10:15:31 -06:00
DottaandPaperclip	86814c6550	fix(recovery): ready retained sandboxes before preparation (#18934 ) ## Thinking Path > - Paperclip manages work performed by AI agents. > - Restart recovery must preserve each run's workspace and execution owner. > - A retained remote lease can refer to an archived sandbox. > - The reattachment path returned that lease without making it ready. > - A short project-authentication command then had to start the sandbox. > - This PR resumes and verifies the exact retained allocation before preparation. > - Recovery keeps the recorded workspace and preserves saved work if resume fails. ## Linked Issues or Issue Description What happened? A native run resumed after a controller restart. Its saved sandbox had been archived. Reattachment returned its lease without a provider lifecycle resume. The next project-authentication probe tried to start that sandbox with its 15-second command budget. The start failed before the probe could inspect project settings. The heartbeat reported a configuration conflict and stopped work. What did you expect to happen? Make the exact retained sandbox ready through the existing bounded lifecycle operation. Verify its allocation before running short preparation commands. Keep the validated lease and recorded working directory. Never acquire a replacement in this path. Steps to reproduce Hold the provider's exact-lease resume pending and call remote reattachment for an active ephemeral lease. Before this change, acquisition returns immediately. The added tests hold acquisition until resume completes and reject missing, changed or expired allocation evidence. Provider compatibility cases retain the recorded workspace when resume omits directory metadata or returns the allocation root. Deployment mode Native Runner in a sandbox with a retained workspace and managed AI connection. The observed provider was Daytona. Refs #18910, which restores selection of retries using current remote workspace storage. This change repairs the next preparation boundary. Related #17421 and #18559 change project-file inspection; they do not make retained sandbox leases ready. ## What Changed - Resume the exact validated sandbox lease before returning it to preparation. - Require the receipt to confirm the original provider allocation without reporting expiration. - Keep the original database lease and recorded directory. Resume metadata can describe the allocation root. - Pass the existing typed resume-directory input through the runtime facade. - Preserve retained E2B sandboxes when resume timeout setup or workspace inspection fails. Keep new-allocation cleanup and explicit release/destroy behavior. - Preserve plugin-task reattachment and lease identity checks. - Update the Stop-accounting fixture to confirm exact-allocation resume before cleanup, with its budget, ownership and no-paid-work assertions intact. - Document the ordering and add readiness, receipt, failure and ownership regressions. ## Verification - Red: seven initial retained-lease cases failed before the readiness change. Two receipt-compatibility cases and four E2B preservation cases failed before their fixes. - Green: all four runtime, workspace-preparation, heartbeat-recovery and orchestrator suites pass: 442 tests. - The Stop-accounting CI failure reproduced locally. All five integration cases pass with the exact-allocation resume fixture. - All 22 E2B tests, the standalone plugin typecheck and its build pass. - 13 Daytona safety cases pass, including recorded directories, allocation sentinel rejection, saved-file reads and no replacement creation. The SDK boundary is simulated; the saved-file case uses real files and a POSIX shell. - These focused checks cover 482 passing cases. - Full recursive typecheck, final server typecheck and full workspace build pass on macOS arm64 with Node 24.22.2 and PostgreSQL 18.1. - Greptile is 5/5 on head `32f915a475275680bcf2a2aa43ae0c72f4e4b1db`. Both original review threads are resolved. - All current-head CI gates pass. The check rollup has 54 successful checks, three skipped checks and one neutral check. - The separate Paperclip Review check remains queued. Its public response says work is paused and the request is saved. The review task is waiting for saved-file recovery, so no Paperclip assessment of this head is claimed. The branch-required gate has passed. - A read-only deployed trace showed an archived sandbox start fail at the short probe's 15-second deadline. Actual provider reattachment with this revision has not been exercised. Live recovery remains unverified until rollout. ## Risks Recovery now makes one bounded lifecycle call before short preparation commands. A missing resume capability or a missing, changed or expired allocation receipt holds recovery. The existing runtime and provider retain allocation and sentinel verification. The path does not acquire, release, replace or reseed the lease. Warm providers can return their existing allocation. E2B retains the existing sandbox if resume setup fails; explicit lifecycle cleanup still owns its removal. No background polling, credential-policy change, migration, workflow change or lockfile change is added. ## Model Used OpenAI GPT-6 through Codex, with reasoning, source inspection, tool use and code execution. The exact backend snapshot and context window were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-04 09:04:13 -06:00
DottaandPaperclip	84e9c8f364	fix(retry): wait for admitted source cleanup before dispatch (#18898 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Connected-task Retry keeps the original request and its current authority. > - One durable Retry can already own a queued successor. > - The old native environment can still have pending cleanup before that successor starts. > - The task execution lock points to the successor, so the old cleanup can escape the queue claim check. > - Dispatch then refuses the source state and fails the new run before provider work. > - This pull request keeps that same run queued and wakes dispatch when cleanup changes. ## Linked Issues or Issue Description Refs #18845, #18853, #18860, #18877, #18882 and #18932. These repairs cover preservation binding, interrupted preparation, old native ownership, manual native holds or other setup guards. This change covers an already admitted connected-task Retry whose native predecessor still owes cleanup. It uses the shared cleanup predicate so the manual-hold repair remains with its existing owner. Search found no duplicate queued-source cleanup repair. What happened? A board Retry saved its exact chat request and admitted one queued successor. The source lease was still pending cleanup when that successor reached dispatch. A retained stop receipt did not settle the full cleanup contract. The queue checked the task's current execution owner, which was already the successor. The native authorization guard then rejected the predecessor state and failed the new run before provider dispatch. Expected behavior Keep the same canonical Retry and queued run while the exact source cleanup is pending. After authoritative settlement, submit that run through normal dispatch once. Do not infer safe execution from a stop-and-retain receipt alone. Steps to reproduce 1. Retry an authorized connected request from a failed native run. 2. Admit one successor while capacity keeps it queued. 3. Leave or restore the source lease to pending cleanup with a matching retained stop receipt. 4. Release capacity and resume the ordinary queue. 5. The old claim path starts the successor and the native source guard refuses it. Paperclip version or commit Reproduced against master at `f77499283270d2e966aabd58493f9d35c6bbdf5f`. Deployment mode The regression uses the real HTTP route, chat authorization, queue claims, disposable PostgreSQL and the production work coordinator on macOS arm64. A controlled adapter supplies the final result. No hosted provider replay or deployment is claimed. ## What Changed - Validate the exact canonical Retry, chat action, company, task, agent, source and successor before testing source cleanup. - Keep the successor queued under the existing issue claim lock while the shared cleanup predicate remains pending. - Wake only the scoped queued-dispatch worker after cleanup or exact workspace-owner release commits. Add no timer or background scan. - Verify pending cleanup at intake, later cleanup invalidation, one final execution, idempotency, Stop, changed input, unknown stop receipts, a different live controller and signed sandbox absence followed by owner release. - Document the queued-wait contract. No schema, endpoint, credential, model or provider policy changes are required. - Align five native preparation acceptance fixtures with the existing workspace publication contract. Use the real reservation and publication owner. Retain a raw-root refusal case with no provider launch and preserved files. ## Verification - Latest master sync at `e26cad76b8f20ba823d702ed60e7b7294f8f5a7c` includes master `396b7fc62a0a4b1d8aadbda8221c05879d54177c`. The shared test helper preserves #18944's canonical retained-directory default, the real publisher for the five fresh-workspace cases, and the explicit raw-root refusal. Independent remerge review found no dropped upstream behavior, guard or assertion. - The latest head passes the focused six cases, all 580 complete workspace/continuation tests, all 38 selected HTTP Retry cases, fresh full recursive Node 24.22.2 types and full build. No affected tests skipped. An initial cached test-layer attempt read old budget declarations; regenerating them resolved all seven errors without test/mock or production-export changes. - All current-head CI is green at `e26cad76b8f20ba823d702ed60e7b7294f8f5a7c`, including required [ci / verify](https://github.com/paperclipai/paperclip/actions/runs/41846304017/job/126636751314) and [ci / e2e](https://github.com/paperclipai/paperclip/actions/runs/41846304017/job/126636638086). Greptile and Paperclip Review both rate this exact head 5/5, with no findings or open review threads. - Master sync at `3e4f266c2da172a84afd0b9cb4a91ea3b95f3c0d` merges master `b891e69bd23d6dab8ce536569e15bc4a7a31c35c`. The only conflicts were adjacent queue imports and two HTTP Retry test matrices. Both imports and both complete matrices remain unchanged. Independent review found no relaxed guard or assertion. - Node 24.22.2 and pnpm 9.15.4: all 38 selected real HTTP Retry cases, 194 queue/cleanup/finalizer cases, module boundaries, recursive types and build pass at `3e4f266c2da172a84afd0b9cb4a91ea3b95f3c0d`. - That sync head's [server check](https://github.com/paperclipai/paperclip/actions/runs/41838741852/job/126604742761) failed five native acceptance cases in CI and locally with `remote_workspace_published_path_invalid`. The test fixture supplied a raw directory instead of a fresh publisher-generated workspace. The guard from #18932 remains unchanged. The same five cases passed earlier Linux CI before that guard was present. - The test-only alignment at `dcdee1bb6c2e2e6dd494382499d17c23a3ce71a8` obtains the saved authority, path and run reference from real `reserveRemoteWorkspace` and `prepareRemoteWorkspace`. All five original cases and a raw-root no-launch/no-result/files-preserved control pass. Their native result, file, catalog and single-execution assertions remain intact. An initial setup attempt was refused for a noncanonical allocation root; the fixture now resolves that owned root before publication. - At previous fixture head `dcdee1bb6c2e2e6dd494382499d17c23a3ce71a8`, the generated server test-layer typecheck passed. All 580 tests in the complete workspace/continuation family pass, with no skips. That test-only commit left production files byte-identical to the sync head. - Previous published-head CI was green at `dcdee1bb6c2e2e6dd494382499d17c23a3ce71a8`, including required `ci / verify` and `ci / e2e`. The [Linux Node 24.22.2 server shard](https://github.com/paperclipai/paperclip/actions/runs/41842665765/job/126618334638) executed all five original acceptance cases and the new raw-root refusal case successfully. Its whole workspace recovery file passed all 245 tests. - Greptile reviewed previous published head `dcdee1bb6c2e2e6dd494382499d17c23a3ce71a8` at 5/5, with no findings or open review threads. No unchanged-job rerun was used for the reproduced sync-head failure. - Causal red: the real HTTP regression started and failed the successor at the native source-state guard while cleanup remained pending. A second red proved that signed sandbox absence followed by copyback-owner release lacked a queued-dispatch wake. - Green: seven HTTP/PostgreSQL cleanup cases, the exact confirmed-owner release notification control and seven existing retry lineage controls pass. The real coordinator resumes from committed cleanup or workspace-owner release, without a manual queue restart. One controlled adapter execution produces the result after settlement. - Earlier branch verification passed full repository `pnpm -r typecheck`, `pnpm build`, module boundaries and `git diff --check`. - The full stale-queue file control passed all 64 cases with unchanged observer/hook limits. The initial broader affected run had 192 passes and one non-assignee native-owner case miss its existing five-second adapter observer while already running with no error. The isolated case passed; the unchanged base passed all 64 cases but hit its existing 30-second teardown timeout. Those original results are retained. The new queue hint has no dispatch consumer in that fixture. - Earlier-head CI was green: 54 success, three skipped and two neutral checks at `6adb9f1ba682282c512d5e751ba2aa663f614ca2`. One chat job was cancelled by an external runner shutdown; its single unchanged-job rerun passed. No source was changed to obtain that result. - Greptile and Paperclip Review both rated earlier head `6adb9f1ba682282c512d5e751ba2aa663f614ca2` 5/5 with no findings or unresolved threads. These results do not qualify the master-sync head. - At initial verification, a complete local `pnpm test:run` was not repeated while the related setup repair's full local run was active. The full earlier-head CI suites passed. A full local suite has not been repeated for the mechanical master sync. Local verification uses the affected HTTP, producer, lineage and queue suites. No full local-suite pass is claimed. - The fresh clone's missing plugin SDK links were repaired with the repository's own postinstall helper before verification. The embedded PostgreSQL package's own symlink helper was also required. - No staging task was submitted again with this repair, and no deployment is claimed. ## Risks - A pending source cleanup can keep the queued run waiting. Its cleanup owner still owns reconciliation and its existing limits. - Stop, changed source, unknown termination and live ownership retain their existing authority checks. A cleanup notification grants no execution authority. - A new cleanup change after claim still faces the existing final dispatch guard. - Cleanup notifications can change dispatch timing in a running controller. The real coordinator cases verify commit-bound wakeups; the broader current queue-file control passed all 64 cases. The earlier loaded-run and baseline teardown failures are retained in verification. ## Model Used OpenAI GPT-6 through Codex, with reasoning, code editing, tool use, PostgreSQL and browser verification. This session does not expose a more specific deployment model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-04 08:33:39 -06:00
DottaandPaperclip	5d226ca08a	perf(chat): reuse bound-run authorization query templates (#18966 ) This PR depends on #18969. Merge that fixture-only prerequisite first, then retarget this PR to master. No merge or deployment is requested in this work. Its base is `codex/fix-recovery-published-workspace-fixture`; the six-line recovery fixture repair is excluded from this diff. ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - External chat authorization binds each run to current messages and conversation policy. > - Review wait and attention-retention paths repeat the same four-table read. > - Each call rebuilt the Drizzle query even when its SQL shape stayed the same. > - This pull request reuses the compiled template for the exact database handle and lock mode. > - Fresh parameters and current database reads preserve authorization while reducing Node CPU work. ## Linked Issues or Issue Description What existing behavior does this improve? The `authorizeChatConversationForBoundRun` links read used by external chat waits and attachment reuse. Current behavior At baseline `e54f9fbf828e06e385e0c62dc381ccb46190aa30`, each call rebuilt and serialized a four-table query. The observed staging shape accounted for 16.2% of sampled SQL serialization CPU across 32 samples. This is a share of query compilation CPU, not total server CPU. Proposed behavior Compile each read, blocking-lock or NOWAIT variant once per exact live database or transaction object. Bind new company, issue, agent, provider and comment IDs on every execution. Continue all current database reads and authorization checks. Reason and benefit Repeated authorized service calls used 26.91% less Node process CPU in the matched local fixture. All seven CPU pairs improved. This reduces repeated compilation on the shared service path. Related: #18896 retains task Retry intent through identity contention. It changes a different workflow and does not reuse this query template. A public issue and PR search found no duplicate template change. ROADMAP.md has no conflicting planned feature. ## What Changed - Keep compiled queries in a `WeakMap` keyed by the exact supplied database or transaction object. Create only the requested lock variant. Never cache a result or authority decision. - Bind the comment list as `ANY(uuid[])`. This supports changing list sizes with only three templates per live handle. Keep all joins, selected fields, policy checks and lock clauses. - Add real PostgreSQL regression tests, a reusable fixture, an actual service benchmark and a dated performance note. ## Verification - Passed 17 real PostgreSQL tests: `node node_modules/vitest/vitest.mjs run --project @paperclipai/server server/src/__tests__/prepared-chat-authorization.integration.test.ts server/src/services/native-runtime/chat-attachment-reuse.test.ts`. - After rebasing, seven selected existing external-chat-wait integration cases passed (266 deselected). They cover small current review waits, answered-chat identity advisory locks, connection-disable serialization and contended endpoint proof retry. - New tests cover changed parameters and list sizes, duplicate/empty inputs, invalid UUID errors, company/agent/provider binding, current setup generation and credential fingerprints, root/transaction/savepoint isolation, rollback, read mode during a held row lock, blocking timeout and NOWAIT contention. - Actual service benchmark: `node --import ./server/node_modules/tsx/dist/loader.mjs server/scripts/benchmark-bound-chat-authorization.ts --baseline-ref=e54f9fbf828e06e385e0c62dc381ccb46190aa30`. Node 24.22.2, macOS arm64, real PostgreSQL, 500 warmup calls per implementation, seven alternating pairs of 500 calls. Every result matched. - Pinned measured treatment `8386eb78d118faaf6c2fd7f7471b38c0f4b2ef54`. Its source is unchanged after rebasing onto master. Median Node CPU per 500 calls: 379.033 ms to 277.048 ms, down 26.91%. Every CPU pair improved by 13.54–33.52%. Median wall time: 742.109 ms to 648.005 ms. One wall pair increased by 0.51%. - A separate diagnostic after timing found five SQL compilations per baseline call and four per warm treatment call. All five database reads still run. - The final four-test PostgreSQL authorization suite passed again after rebasing and strengthening the lock proof. It holds only a message-link row, so principal advisory contention cannot mask a missing joined-query lock. - Independent source review found no correctness issue in exact-handle ownership, fresh binding or the three lock variants. - Preserve setup failures: the first test attempt lacked plugin SDK build output. Shared and SDK builds fixed it. Later attempts could not start embedded PostgreSQL because its local dependency directory lacked bundled native-library aliases. `DYLD_LIBRARY_PATH` did not fix it. Restoring package-local aliases fixed the environment; the final 17 tests then passed. No dependency or lockfile change is included. - The first full typecheck found new test-only errors: PostgreSQL transaction handles lack the root database `$client` type, and runtime setup test data includes fields outside the saved setup interface. Commit `40a4645750` uses the existing explicit transaction test cast and SQL JSON fixtures. No production contract changed. The full check also found local plugin SDK link prerequisites, which the coordinator is repairing separately. - Preserve the first Linux CI failure: [server shard 9](https://github.com/paperclipai/paperclip/actions/runs/41837775564/job/126602280304) failed the new setup-fence fixture's allowed case. The fixture supplied a client `processedAt` while PostgreSQL supplied default `receivedAt`; this could violate the required processing order. The test now uses explicit ordered timestamps and no wall-clock window. The four-test PostgreSQL suite passed again in 19.93 seconds. Production guards were not weakened. - [Server shard 2](https://github.com/paperclipai/paperclip/actions/runs/41837775564/job/126602280291) also failed five workspace-recovery cases with `remote_workspace_published_path_invalid`. Those failures reproduced on both pure master and the exact CI merge with the chat optimization removed. #18969 fixes the canonical path in that separate fixture, with five failing cases before and five passing after. This PR is stacked on that independent repair for final CI. - The additional Paperclip review reported a node-postgres statement-name collision and requested explicit pool cleanup. This repository uses postgres-js; locked Drizzle 0.45.2 accepts the Drizzle label but does not use or forward it. [The review response](https://github.com/paperclipai/paperclip/pull/18966#issuecomment-6632666599) links primary driver source and the database factory. Commit `c5d0d84f36` adds explicit pool teardown in `try/finally` and a real same-transaction `read → blocking → nonblocking → read` case. All five authorization tests passed in 6.47 seconds. The original separate row-lock test remains. Production SQL and code did not change. - Final combined local validation at `4ab58ff70b` passed full `pnpm -r typecheck` (39.864 seconds) and 35 real PostgreSQL cases across the chat, GitHub and shared prepared-query suites (11.378 seconds). Full `pnpm build` passed earlier with byte-identical production source; later changes affect tests only. The production file SHA matches the benchmark and this PR. The original full `pnpm test:run` was interrupted after all six chat shards passed and general tests had partly run, so it is not a full local pass. Final current-head CI provides the broader coverage. A sandbox retry failed with EPERM before assertions; the identical escalated commands then passed. Final head passed CI in [workflow 41843806060](https://github.com/paperclipai/paperclip/actions/runs/41843806060), including all build, typecheck, general, serialized and e2e test gates. The previous head `40a464575009dfc3cce7afa1eb4fffa5540dcaf6` passed Greptile 5/5, security review, typecheck, build and canary validation. The prior stacked head `74cb36370162750f8e135ed091603d3629d23b0d` passed full CI, Greptile 5/5, policy and security checks. Final head `c5d0d84f369d897a66d9b6441bab10cb79f593b9` passed a fresh Greptile 5/5 review with no actionable findings, security and PR policy checks. Its complete [current-head CI workflow passed](https://github.com/paperclipai/paperclip/actions/runs/41843806060). All 54 required CI, Greptile, policy and security checks passed. The optional Paperclip Review check is queued for this head; no current-head score is claimed. Production source SHA-256 remains `72ce9d2317e5c97b36e299e24e30fc1aa63020d8c266e495fc45dbfdd5331e93`, identical to the measured code. ## Risks - `ANY(uuid[])` relies on postgres-js array binding and UUID casting. Real PostgreSQL tests cover single and multiple valid values and invalid UUID errors. - A new transaction or savepoint pays its first compilation cost. The weak cache cannot reuse a root session for it. This benchmark covers repeated calls on one root handle. - The team paused other work during the benchmark. Unrelated Rust compilation was active elsewhere on the host. Process CPU excludes those processes, but host contention can affect caches, scheduling and wall time. This source-mode local result does not prove production request latency or whole-server CPU savings. - No migration or API contract changes. No deployment or merge is included. Monitor authorization failures and lock-contention behavior after an authorized rollout; revert the narrow service change if those regress. ## Model Used OpenAI `gpt-6.1-sol` through Codex, with reasoning, repository editing, tool use and test execution. The host did not expose a context-window size or deployment snapshot ID. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-04 08:32:31 -06:00
DottaandPaperclip	2cbfc35a07	perf(github): reuse the inherited review wait query template (#18965 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - External review waits check current GitHub review and publication bindings. > - The GitHub service rebuilds the same seven-table Drizzle query on each check. > - A sampled production window attributed 6.3% of SQL serialization CPU to this query shape. > - This pull request prepares that query once per exact database or transaction object. > - Each call still reads current provenance, review, connection and publication state. > - The local service benchmark reduces median client CPU per read by 24.35%. ## Linked Issues or Issue Description What existing behavior does this improve? Repeated `readInheritedGitHubReviewWait` calls from external review wait reads and attention retention. Subsystem affected Server GitHub chat review waits. Current behavior Every call constructs and serializes the same joined SELECT. The database driver already supports prepared statements. That setting does not reuse the JavaScript query template. Proposed behavior Compile the existing selector once for each supplied database, transaction or savepoint object. Bind company, issue, agent and comment IDs on each execution. Keep every current database read and check. Reason and benefit Remove repeated query construction and serialization CPU. Do not cache rows or authorization decisions. Exact open-PR searches found no duplicate query-template change. ## What Changed - Add an exact-handle `WeakMap` for the GitHub review selector and execute it with fresh placeholders. - Preserve the SQL projection, joins, JSON types, latest-review order and all current authority checks. - Extend real PostgreSQL tests for new bindings, warm-cache revocation, latest heads and transaction/savepoint rollback. - Capture prepared executions in the existing test helper. - Add an actual-service benchmark and a report with every timing sample and setup failure. ## Verification - `github-review-wait.test.ts`: 16 tests passed on real PostgreSQL. - `external-chat-wait.integration.test.ts -t 'reads small current external review waits without saved prompts'`: four affected cases passed; 269 unrelated cases skipped. - Pinned baseline: `e54f9fbf828e06e385e0c62dc381ccb46190aa30`. Final source SHA256: `fe647e7fe2aed82609e8f1d66a0109e7a10ee28313ae2ef218ba3ab39b5cd105`. - Exclusive-window benchmark: 500 warmups per version, nine alternating pairs of 100 complete service reads on disposable PostgreSQL. Median client CPU: 1.00336 → 0.75908 ms/read (24.35% less). Median wall time: 2.87089 → 2.38731 ms/read (16.84% less). CPU excludes the PostgreSQL process. - Every generated SQL statement, parameter, query count and complete result matched. Warm joined-selector compilations per ten service reads: 10 → 0. - See `doc/performance/2026-11-04-prepared-github-review-wait.md` for reproduction, all samples and measurement limits. - Setup record: loopback required execution outside the sandbox. The fresh embedded-Postgres dependency lacked macOS dylib aliases; aliases to existing identical bytes restored the fixture. The SDK import needed a current workspace SDK/shared build. A broad external-chat suite run was interrupted and replaced by the affected test filter. Those attempts are not counted as passing suites. - Greptile: 5/5 on current head `d2e6ae8814c3a56c380bd5d60c47e6f98642f085`, with no actionable issues or inline comments. - Current-head CI is green at `d2e6ae8814c3a56c380bd5d60c47e6f98642f085`: 52 successful checks, three intentional skips and the successful Snyk status. This includes typecheck, build, the full test matrix, end-to-end shards and the aggregate gate. - Combined local validation at `4ab58ff70b` includes both measured production templates and final test fixes. Full recursive `pnpm -r typecheck` passed in 39.864 seconds. Three real PostgreSQL files passed 35/35 tests (five chat cases, 16 GitHub review cases and 14 shared prepared-query cases). The combined build passed earlier with the same production template bytes. - The initial combined typecheck failed on a separate chat test typing error and build setup. The later passing run resolves that attempt. The original full local `pnpm test:run` was deliberately interrupted after six chat shards passed and partial general tests ran; it is not a passing full local suite. Each PR has independent current-head CI for broad coverage. The initial loopback sandbox restriction was resolved by an escalated run of the same source. - Local benchmark/test dependencies were locked at the baseline; the rebase adds the upstream lockfile refresh and UI paste fix only. ## Risks - A prepared query captures its session. The exact supplied handle is the cache key so transactions and savepoints do not borrow the root session. - Placeholder mistakes could cross bindings. Tests alternate independent company/issue/agent/comment fixtures on the same handle. - The benchmark has one synthetic eligible fixture with warm sequential reads. It does not predict production endpoint latency or whole-server CPU savings. - No schema, index, database driver setting or public API changes. ## Model Used OpenAI `gpt-6.1-sol` through Codex. The context-window size is not exposed by the session. Used reasoning, tool calls, source editing and code execution. Independent Codex review found no correctness issues. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-04 08:29:59 -06:00
DottaandPaperclip	0eaabe5c0b	refactor(accounting): remove company serialization from paid settlement (#18917 ) Settle paid operations through operation-scoped transactions. Preserve exact ledger evidence and replay while moving aggregate budget decisions to durable deferred work. Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-11-04 07:29:28 -06:00
DottaandPaperclip	669538534c	fix(recovery): preserve failed remote reattach workspaces (#18936 ) ## Thinking Path > - Paperclip manages AI agents and their work. > - Native runs can edit files in a remote execution workspace. > - A failed reattach can leave an observed coordinator and unpublished files. > - The failure handler recognizes the older workspace-sync descriptor but misses the modern workspace reference. > - The released task lock then prevents ordinary session recovery from claiming that old run. > - This pull request retains the files through the existing Stop, cleanup and workspace recovery owners. > - Exact ownership and Stop evidence permit file preservation while unknown provider effects remain under manual reconciliation. ## Linked Issues or Issue Description Related: #18877 protects native manual admission holds. #18879 handles scheduled native retry lock reclaim. #18918 fixes remote workspace row-lock order. Those changes do not record the missing modern failure-retention obligation. What happened? A native reattach failure can leave a failed run, a dirty modern remote workspace, an expired observed coordinator and a free task execution lock. The run has no accepted result. The failure handler skips file retention because it checks only the older descriptor. Recovery cannot export while the old coordinator remains owned. A mismatched workspace generation can also make orphan cleanup miss the existing file claim. Expected behavior Retain the exact original allocation and files. Stop the allocation through the existing lifecycle owner before retiring the stale coordinator. Keep checkpoints and unknown external effects available for manual reconciliation. Unsafe or changed source authority must prevent physical cleanup. Steps to reproduce 1. Prepare a modern remote workspace for a native run. 2. Record a provider checkpoint and an active interruption incident. 3. Fail reattach preparation and release the task execution lock. 4. Let the coordinator lease expire. 5. Run the existing orphan, environment cleanup and workspace recovery workers. 6. Verify the original files from a cold archive read and verify that no new native session starts. ## What Changed - Include modern prepared references in the existing heartbeat failure-retention producer. - Add a file-only historical intake for an exact failed observed run in the normal expired reattach state. - Require an untouched active ephemeral lease, a free blocked task, matching workspace writer and generation, no accepted result and complete single-owner allocation history. - Persist the existing StopAndRetain intent before provider I/O. Use nonblocking row locks and classify busy ownership only after rollback. - Use the existing native environment settlement owner after exact Stop and retention receipts arrive. Keep its single scoped discovery query. - Preserve source and coordinator fingerprints across the handoff. Keep original checkpoints, errors and manual unknown-effect incidents. - Keep stale or malformed modern workspace claims held when preservation intake refuses them. - Add actual PostgreSQL, heartbeat, allocator, worker restart, archive and safety controls. Document the narrow recovery contract. ## Verification Current published head `b1f49691187a398415cf9b16b723e3ab473f0f91`: all required CI checks pass (54 success, 3 skipped, 1 neutral); Greptile is 5/5 with no open threads. The separate Paperclip assessment is queued and its current response says work is paused. That assessment is unavailable; no additional review nudge or hosted retry was issued. All local commands use Node 24.22.2 on macOS arm64. - Exact published head: `b1f49691187a398415cf9b16b723e3ab473f0f91`. Required CI passes (54 success, 3 skipped, 1 neutral); Greptile is 5/5 with zero open threads. The optional Paperclip assessment remains queued/paused without a completed score. - Final recursive `pnpm -r typecheck`: passed. - Final `pnpm build`: passed. - Final three native producer, containment and scheduling files: 67/67 passed. These include the real coordinator and worker restart, the production heartbeat recovery callback, exact Stop receipts, cold archive restore, and zero native backend, new allocation or destructive release calls. - The plugin allocator cases use the actual selector with `reuseLease: true`. Active and released ephemeral leases receive a fresh allocation. - The assembled reattach producer checks that the task and coordinator claim commit before physical resume. A historical pin refuses that current owner. A later claim behind a committed pin remains held. - The safety controls cover reusable allocations, manual and unknown states, cleanup in flight, accepted but unbound results, late allocation owners, wrong or malformed workspace authority, changed generation/checkpoint, lost Stop acknowledgement and a real cleanup-producer lock conflict. - Untouched base `e7549ab6131555f655736e6b544892a3b1e323cb`: four expected behavioral failures and one passing legacy-descriptor control. The failures cover the modern producer, original provider history, historical retention and wrong-epoch cleanup. Fixed production files were restored byte-for-byte after this comparison. - The first affected six-file run completed 562 passed / 1 failed. The failure was an existing HTTP case with `socket hang up`. That exact case passes separately on both untouched base and this branch. The original connection failure remains unexplained. At `0e92de8c`, the complete workspace family passes 188/188 with unchanged assertions. The environment cleanup owner family also passes all 23 cases. The later guard changes are qualified by the complete 67-case native owner group. A full repository test run was not repeated locally; the requested full-ci workflow runs the complete repository suites. - Published `0e92de8c` fails both new explicit-null Stop/export marker controls because it creates a new retention pin. The follow-up uses the shared presence-aware Stop guard. Both controls now preserve the marker and record no new retention pin, Stop or destructive release. - Paperclip's review found that malformed workspace authority could throw and abort an orphan page. Published `5f6cd4f5` fails all three new controls: intake, the actual page and after-Stop settlement. The same strict schema now uses a safe parse, and the existing OBSERVED hold runs before the prepared-source lookup. The malformed source stays held while the next valid source receives its retention pin. - Two existing scheduling tests caught the draft's second global discovery query. The final single-query repair passes all 36 unchanged scheduling tests. - An earlier restarted archive assertion stopped after 20 seconds. Its recorded Stop was valid, and the coordinator had settled. The existing evidence recheck deadline was still 13 seconds ahead. The final test observes the unchanged 24–36 second policy within its existing 60 second case limit. The external fake provider Stop and file transport are controlled test boundaries. Database ownership, lifecycle workers, archive bytes, encrypted local storage and cold restore are real. This does not claim a live cloud sandbox, S3 or model run. No hosted task was changed for this PR. ## Risks - Historical intake is deliberately narrow. Reusable, already released, mismatched or competing allocations remain held. - File preservation does not settle unknown provider effects or authorize a repeated provider turn. The task can still require manual reconciliation. - A busy or changed owner delays preservation. The original files and claims remain retained. - This does not repair the underlying sandbox readiness failure or establish the cause of a separate archive-capture failure. - Deployment and recovery of original hosted tasks remain unverified. ## Model Used OpenAI Codex, GPT-6 family, with repository inspection, code execution and GitHub tools. The runtime does not expose the exact serving model ID or context window size. A separate Codex agent and the coordinator performed read-only source review. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs: #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-04 07:19:03 -06:00

1 2 3 4 5 ...