mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-08 12:33:05 +01:00
21640b6d514a5569cd76b2d3a3bb9e13e2a623d2
511
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 DottaandClaude Opus 5.5
|
5498db1072 |
fix(interactions): relay a verified reply to a self-requested human-only confirmation (#19243)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents ask users to approve proposals and plans with confirmation cards, and users often answer in the conversation > - `resolve-from-comment` lets the responding agent record that verified reply on the card > - Native agents often create their own proposal or plan confirmation as `human_only`, because the `request_human_input` schema offers only that value > - `resolve-from-comment` then refuses with `interaction_human_only`, and the plan and onboarding gates keep the task in `in_review` or `blocked`, although the user explicitly approved > - This pull request adds one narrow relay for that case and keeps every other human-only gate on its card > - The benefit is that an explicit conversational approval works again, and governed, company-capped and quality-review approvals keep their existing controls ## Linked Issues or Issue Description Refs #19073 **What happened?** TestOps launch https://paperclip.testops.cloud/launch/2694 (GitHub run https://github.com/paperclipai/paperclip/actions/runs/41915965118) failed three Product regression cases for the same reason: `continuation/revision-preserves-approval`, `first-task/clarify-propose-accept` and `first-task/revise-accept`. In each case the agent created a `request_confirmation` with `resolverPolicy: human_only` (`effectiveResolverPolicySource: requested`). The user replied "I approve…" or "Yes, I accept that proposal". The agent's `resolve-from-comment` call returned 403 `interaction_human_only`. The task stayed `in_review` (plan gate) or became `blocked` (onboarding decision gate). **Expected behavior** When the agent itself asked for a human-only answer, the verified latest reply of the user it serves is that human answer. The agent can record it. Company-capped, governed-action, issue-review and quality-review gates still require the human to use the card. **Steps to reproduce** 1. A native agent calls `request_human_input` with `interactionKind: confirmation` and `resolverPolicy: human_only`. 2. The conversation user posts an explicit approval comment. 3. The agent calls `POST /api/issues/{id}/interactions/{interactionId}/resolve-from-comment`. Before this change it gets 403. ## What Changed - `issue-thread-interaction-resolution.ts`: the agent actor gets an optional `relayedHumanUserId`. A human-only decision is allowed as `allow_relayed_human` only when all of these are true: - the stored policy is `human_only`, its source is `requested` and its provenance is `explicit` - the resolving agent created the card (`createdByAgentId` matches) - no governed action applies - no additional restriction (issue review) applies - any named addressee user is the relayed user Company caps, the `governed_action` source and legacy rows with an unknown source are still denied. - `confirmation-comment-resolution.ts`: the service sets `relayedHumanUserId` only after its existing comment checks pass: user-authored, not agent-attributed, posted after the card, the latest user message, the same conversation session. It also requires that the author is the card's addressee, the conversation user or the run's responsible user, that the card is not a quality review, and that the company's current cap for the kind is not `human_only` (a tied cap keeps the source `requested`). The same actor goes into the locked accept/reject re-check. The activity entry adds `resolverPolicy: "human_only"` and `resolution: "relayed_human_decision"`. - `routes/issues.ts`: the route pre-check passes the candidate comment author. The service still re-verifies everything under its locks. - `issue-thread-interactions.ts`: carries the relay field into the service's own audience re-check. - Docs: the first-task skill and `api-reference.md` describe the narrow relay. The guidance still says not to add `human_only` to ordinary proposals. ## Verification - `server/src/__tests__/confirmation-comment-resolution.test.ts` (embedded Postgres): 72 pass. - Green: relay through a conversation user and through a run's responsible user. The card is `accepted`, `resolvedByAgentId` and `resolvedByRunId` are set, `resolvedByUserId` stays null, and the audit records `relayed_human_decision`. - Negative controls, each with the card still pending and no audit: agent-authored comment, earlier comment, wrong user, unbound issue/run, `company_cap` source, `governed_action` source, `qualityReview`, `toolAction`, `connectionAuthorization`, issue-review `human_only` restriction, another addressed user, human-created card, another agent's card, inherited company default, and a real capped card created through `svc.create` with a tied cap. The tied-cap test fails as expected when the cap is removed. - Existing negative controls are unchanged, including `human_only` without a bound user and the "rechecks narrowed permissions" retry. - Red check: with the three service files reverted to `master`, the two new green tests fail and the negative controls pass. - `server/src/services/issue-thread-interaction-resolution.test.ts`: 21 pass, including the pure evaluator relay rules. - `server/src/services/native-runtime/confirmation-comment-api.integration.test.ts`: 11 pass. The real route accepts the relayed reply and refuses another user's comment with 403. - Also pass: `issue-thread-interaction-routes`, `onboarding-first-task-admission`, `paperclip-runner-tool-authority`, `runner-api`, `onboarding-first-task-assets` and `company-skills-service` (2,474 tests in total). - Changed files typecheck cleanly. - Live verification: Product regression single-case runs of the three cases are pending after merge. ## Risks - This touches an approval gate. The relay is limited to cards whose creator chose `human_only`. All existing reply-evidence checks still apply, plus a bound-user requirement. Company caps, governed actions, quality reviews, issue-review restrictions and named addressees still deny the relay. - Attribution does not change: the resolver is still the agent and run. The human's comment is kept as `result.commentId` and in the audit. - Legacy rows with a null policy source are not relayed. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
26391707fb |
fix(runner): never settle an OpenCode reply truncated at the output limit (#19242)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native OpenCode runs turn the provider's last assistant text into
the task's visible reply
> - A model can loop until it reaches the output limit. The provider
then reports `finish: "length"`
> - The driver used that cut-off text as the final answer, and the run
succeeded with a broken reply
> - This pull request treats a response that stopped at the output limit
as incomplete
> - The benefit is that users get the accepted completion summary or one
corrected answer, never a truncated wall of text
## Linked Issues or Issue Description
**What happened?**
In a QA staging run, an OpenCode agent on
`openrouter/deepseek/deepseek-v4.1-flash-1017` called `paperclip_finish`
correctly. Then it produced about 32,000 output tokens of repeated text
(`E2E_QUIP_QUIP_...`) until the output limit stopped it. The driver
selected this truncated text as the settled final answer. The run
succeeded, and the task reply was wrong.
**Expected behavior**
A response that the provider stopped at the output limit is not a
complete answer. If an accepted completion report exists, its summary is
the reply. If not, the driver asks for one concise final answer. If that
also stops at the limit, the turn fails.
**Steps to reproduce**
1. Run a native OpenCode turn where the model calls `paperclip_finish`.
2. Let the next assistant message run to the output limit (OpenCode
reports `finish: "length"`).
3. Before this change, that message text is emitted as the final
`agentMessage`.
**Paperclip version or commit**
master at
|
||
|
DottaandPaperclip
|
47027e7ba5 |
fix(routines): retain authenticated agent responsibility on creation (#19222)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Routines save work for later execution under a responsible person's authority. > - An agent can create a routine through its normal REST or native API tools. > - The create route kept the agent creator but dropped the authenticated responsible user. > - The service then selected a parent task's user or a company default, even when the source run had a different user. > - This pull request passes the authenticated responsible user separately from creator attribution. > - Future automatic runs then use the authenticated responsible user saved in the routine revision. ## Linked Issues or Issue Description **What happened?** An authenticated native agent created a routine without a parent task. The stored responsible user became a different company owner. An unrelated parent task could also replace the authenticated user. The defect changes the identity saved for future automatic work. **Expected behavior** Creation must use the server-resolved responsible user from the authenticated run or API key. The agent remains the creator. A person who clicks Run now still supplies that manual run's identity. **Steps to reproduce** 1. Create an active company member and a different company owner. 2. Initialize a native agent run with the member's identity. 3. Use `call_api` to create a paused, self-assigned routine without a parent. 4. Read the routine and its first revision. Before this fix, both use the company owner. **Paperclip version or commit** Reproduced on master `56517dda6c5b32bc901b7dbf1605eaaa041c5fba`. **Deployment mode** Local source integration with real HTTP routes, agent authentication, native tool authority, and embedded PostgreSQL. No provider turn is needed for the reproduction. Related work: [#19128](https://github.com/paperclipai/paperclip/pull/19128) adds board controls for saved responsibility. [#15207](https://github.com/paperclipai/paperclip/pull/15207) repairs synthetic identities from built-in installers. Neither fixes this authenticated create path. ## What Changed - Pass the authenticated agent's responsible user as a separate internal actor field. - Select the board user first, then the agent’s server-resolved responsible user, before the existing parent and company fallback. - Keep creator attribution, initial revision attribution, manual dispatch identity, and existing records intact. - Add real native API, HTTP, and PostgreSQL regressions for identity selection, replay, client spoofing, stale claims, revoked membership, cancellation, cross-company calls, API keys, and fallback behavior. - Document the creation and manual dispatch identity rules in the execution contract. Correct the obsolete responsibility sentence in the agent access map. - Make the existing live-retry route fixture wait for its committed comment’s durable wake before draining. Keep all process and saved-owner assertions. ## Verification - Captured a failing native `call_api` test before changing production code. The persisted responsible user was the unrelated company owner. - The same real route and database boundary passes after the fix. - Initial affected routine and identity checks: 131 passed. The existing pending-AI-card fence also passed. - Follow-up routine checks: 118 passed. The owning scheduled-retry suite passed all 31 tests. - Initial CI serialized shard 7/9 failed with a `23503` wake-to-agent foreign key error during test cleanup. A controlled real-route/PostgreSQL pause reproduced that exact error on unchanged base `56517dda6c`. The drain returned before the route registered its deferred comment wake. The same pause passed after the fixture waited for the actual commentId’s durable wake receipt. The original failure is preserved. - Affected server typecheck after the fixture query change: passed. - `pnpm -r typecheck`: passed. - `pnpm build`: passed. - `pnpm test:run`: still running as supporting coverage. The worktree changed during the run: map cleanup, then the six-line fixture wait. Production and routine-test files stayed unchanged. This is mixed-working-tree coverage, not an immutable full-suite pass for a commit. Exact next-head CI remains required. - `node scripts/check-maps.mjs` and `git diff --check`: passed. - Exact head `a12e3f6c42b8ad5826fd58faa3cd9553ea251e09`: all CI gates passed. Greptile and the native substantive review both gave 5/5 with no open findings. The native reviewer did not run tests or exercise shadow-mode authorization. - Live provider execution remains unverified. This change does not grant access to another person's AI account. ## Risks New agent-created routines now use their authenticated responsible user before a parent or company default. This is the intended behavior change. The service keeps its fallback when no bound identity exists. Missing agent-key identity still fails authentication. Missing personal AI configuration still blocks execution. Existing routines, revisions, tasks, grants, and manual dispatch behavior do not change. The fixture waits for a persisted wake record before draining; its original live-process and saved-owner assertions remain. ## Model Used OpenAI `gpt-6.1-sol` with reasoning, tool use, and code execution. The host did not expose the context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
3a090b815b |
fix: retain native checkpoint ownership through warm close (#19225)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native warm session must save its checkpoint under its exact coordinator owner. > - A failed turn can return an accepted result before that checkpoint is saved. > - The executor stopped renewal and released the owner before the managed session closed. > - The backup owner guard then refused the uncommitted, released owner. A failed close could remain in the warm session map. > - This pull request keeps the claim and its existing renewal through checkpoint, close and instruction collection. > - The benefit is a valid checkpoint owner and one fenced release after the work joins. ## Linked Issues or Issue Description Refs #19208, #19206 and #14695. Those fixes cover a destroyed backup source, Stop before a provider session opens, and retained agent-file ownership. They do not change this checkpoint ownership order. **What happened?** A failed warm turn accepted its result. Its coordinator reached `workspace_finalizing`, with a result ID and no lease owner. Managed close then called the durable checkpoint writer. The writer correctly refused this state with `native_harness_storage_owner_missing`. The failed close could remain for the next fresh session to retire. **Expected behavior** The current claim and renewal remain valid until the required checkpoint, close and instruction collection join. Cleanup stops and joins renewal, then releases only that exact claim. A changed owner is not cleared. An earlier error keeps its identity. **Steps to reproduce** 1. Run the new real-Postgres regression against the base commit. 2. Complete a structured failed warm turn through the normal control-plane result producer. 3. Let the actual managed-session close call its production checkpoint callback. 4. Observe the missing-owner refusal after coordinator release. **Paperclip version or commit** Base: `56517dda6c`. The native ownership order also exists on the earlier failing revision. **Deployment mode** Native warm execution with durable harness storage and a managed remote workspace. ## What Changed - Join warm checkpoint, managed close and instruction collection before coordinator release. - Keep the existing lease renewal active through that work. Stop and join it before release. - Run the same fenced release after a checkpoint or collection error. Preserve the initiating error unless restart detachment has won. Keep the typed restart outcome and changed-owner fences. Attach only closed cleanup-failure flags. - Add real-Postgres ownership, delayed-renewal, error, retry and denial cases. Route the managed-session regression through the existing artifact-equipped native test lane. Update the continuity test for the retained claim. Document the active renewal lifetime. ## Verification - Red: the producing coordinator had an accepted result, `phase=workspace_finalizing` and `leaseOwner=null` at managed close. The production checkpoint callback refused it with `native_harness_storage_owner_missing`. - Green: all 9 real-Postgres regression cases pass. They cover a real 20-second renewal tick, a failed checkpoint, one force-fresh successor, and wrong-boot and changed-owner denial. Actual restart detachment wins even when checkpoint and joined renewal-stop both reject. The replacement claim remains unchanged. No new provider turn or cancellation receipt is created. A private canary does not enter the closed diagnostic flags or an error cause. - The executor, restart-recovery and heartbeat compatibility suites pass: 1,259 tests, with 20 gated skips. The container-enabled continuity case was not run locally. Its 21-second timer check remains. It now requires the exact claim during `hasChanges()` and checks release after the joined work. - All 39 shard-routing and combined-coverage tests pass. The new regression runs in the existing native lane with the required Runner artifact. - Final local head `0edcc37f35` passes `pnpm -r typecheck`, `pnpm build`, `node scripts/check-maps.mjs` and the 9-case regression. The earlier broader compatibility run used identical source bytes. A separate earlier repo-wide local suite on `6cdd68e915` ended with exit 1. Its general-server stage had one GitHub enrollment acknowledgement failure, 31,469 passing tests and 215 skipped tests. That acknowledgement failure reproduced in one narrow run on the same old tree. It is not an established flake or proven caused by this fix. Later repository stages did not run. No duplicate broad suite or full local pass is claimed. - The first published head failed review for detach-error precedence and the old continuity assertion. Its server shard also failed because the new test lacked a Runner daemon in the general lane. Those failures remain recorded. This candidate fixes their causes. Both exact-head reviews are now 5/5. The hosted native-server lane ran all 9 new ownership tests successfully. One unchanged tool-gateway rate-rule test failed in the first current-head CI run. It passed previously and passed locally, both narrowly and in its full 183-test file. The single same-head failed-job rerun passed. All current-head hosted gates are clean. The initial failure remains recorded; no test assertion was weakened. - The provider transport is an in-memory fixture. The managed wrapper, production checkpoint callback, control-plane result and Postgres ownership are real. No paid provider or live remote process is exercised. This does not prove a deployed recovery or a remote stop receipt. ## Risks - A slow required checkpoint extends the lifetime of the existing active renewal. Its cadence stays at 20 seconds for remote claims and 5 minutes for local claims. It runs only while the admitted checkpoint, close and collection remain active, then stops and joins on completion or unwinding. The tests observe a real remote renewal tick. No idle timer, new retry budget or new worker is added. - The release still checks company, task, run, owner, attempt, boot, PID, process start and expiry. Restart detachment retains its recovery owner. - The storage owner guard is unchanged. No migration, permission, provider setting or deployment change is included. ## Model Used OpenAI Codex, `gpt-6.1-sol`, with reasoning, source inspection, local code execution and read-only staging metadata. Context window and reasoning effort are host-managed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
beb73381ca |
fix(native): let a Stop before the provider session opens continue once (#19206)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A remote task restores its saved workspace into a sandbox before the agent starts. > - When that restore fails, the run records only `remote_workspace_operation_failed: operation=restore exit=1`. > - The run then fails as `setup_failed`. No retry starts, and a `blocked` task gets no recovery action. > - An operator cannot tell which step failed, and nobody owns the next step. > - This pull request makes the sandbox scripts name their failed step in a closed grammar, and it gives a restore failure a bounded retry or a named board recovery action. > - The benefit is a visible cause and a clear owner. Saved work stays protected. ## Linked Issues or Issue Description Refs #19152 and #19187. Those changes name copy-back (save) failures. Refs #18852 and #19195. Those changes added the bounded bootstrap retry that this change reuses. **What happened?** A board reply woke an agent on a `blocked` parent task. The run failed during setup 13 seconds later. The only run event was `remote_workspace_operation_failed: operation=restore exit=1`. The run had no workspace operation row. The task stayed `blocked`, and its recovery actions showed `active: null`. The same opaque family also appeared for workspace preparation (`exit=1` only). **Expected behavior** The run names which restore step failed, without command output or paths. A failure before provider work uses a bounded retry when a retry can help. Otherwise, and when the retry budget is spent, the board gets a named recovery action. The saved version is never changed or overwritten. **Steps to reproduce** 1. Save a remote workspace, then start a new run on a different allocation. 2. Put a folder at the published workspace path in the sandbox before restore. 3. Observe that the base code records only `operation=restore exit=1`, then `setup_failed`, with no recovery action for a `blocked` task. **Paperclip version or commit** Base: `87dfba6818`. **Deployment mode** Legacy runtime with sandbox (Daytona) remote workspace storage. ## What Changed - The prepare, repository, and restore sandbox scripts track their current stage. On failure they print `remote_workspace_<op>_failed: stage=<stage>` plus closed fields: `reason=` (the script's own snake_case refusal), `archive=` (the archive helper's fixed reason), `code=` (errno), `disk=full`, `git=` (a fixed label chosen from Git output), and the existing `exit`/`signal`/`program` fields. - The server accepts that text only when it matches a strict grammar for the same operation. Other output is dropped, as before. - Provider refusals during the restore upload and restore command are recorded by closed labels, like copy-back. - A failed saved-workspace restore raises `RemoteWorkspaceRestoreError` with the server step (`snapshot_lookup`, `snapshot_download`, `sandbox_restore`). It is retryable only when the owner re-reads the same writer, epoch, and saved version with no unsaved work, and the sandbox did not name a refusal reason. - Setup failure handling classifies a pre-dispatch legacy restore failure as `remote_workspace_restore_failed`. A retryable failure uses the existing bounded bootstrap retry. A non-retryable or exhausted failure records a board-owned recovery action with the redacted cause, when stranded-work recovery does not scan the task status (for example `blocked`). - Stranded-work recovery gets a named notice for this code. Generic continuation recovery does not grant it a second budget, and legacy reconciliation does not invent provider actions for it. - `doc/execution-semantics.md` documents the contract. ## Verification - Red: the new transport tests failed on the base code with `remote_workspace_operation_failed: operation=restore exit=1` (and the same for prepare and repository). The new completion tests failed because the run became `setup_failed` with no retry and no recovery action. - Green: `npx vitest run src/services/remote-workspace-transport.test.ts src/services/heartbeat/run-completion.test.ts src/services/remote-workspaces.integration.test.ts src/__tests__/heartbeat-retry-scheduling.test.ts src/services/legacy-execution-recovery.test.ts src/services/recovery src/services/remote-workspace-save-concurrency.test.ts src/services/native-runtime/native-workspace-finalizer-failure.test.ts src/__tests__/native-workspace-finalization-work.test.ts` passes (563 tests). - The transport tests use real sandbox commands. They prove an occupied folder reports `stage=inspect_root reason=occupied_root` and keeps its files. They prove a wrong manifest reports `stage=extract archive=manifest_mismatch`. They prove a missing branch reports `stage=checkout_branch`, and a changed origin reports `stage=verify_origin`. Forged output with a path is dropped. - The real-PostgreSQL cold-restore test proves an occupied root gives a non-retryable `sandbox_restore` failure and keeps the unknown files. - The completion tests cover retried, exhausted, refused, post-dispatch, and `in_progress` cases. - `tsc --noEmit` for the server reports no errors in the changed files. ## Risks - A retry keeps the same restore path. If the cause is deterministic but unnamed, the run uses its two retries before the board action. Named refusals skip the retries. - The board action is recorded only for statuses that stranded-work recovery does not scan, and only when no other action is active. This prevents a second recovery identity for one failure. - Script stderr grammar is new. Old sandboxes run the script that the server sends, so no runner update is necessary. - No migration, deployment, or credential change. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use, code execution, and browser tools for read-only staging evidence. Context window and reasoning mode are host-managed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
f5891564ec |
fix(recovery): continue from a retired sandbox's backup and hand spent export repairs to the board (#19208)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native tasks keep a warm sandbox and a verified per-turn backup of the provider session. > - Recovery holds stop a task when continuation is not safe. They must have a next action and an owner. > - In a staging soak, two kinds of held tasks could not move forward. Users followed the product's own instruction, and nothing changed. > - First, a reply to a task whose warm sandbox Paperclip had already destroyed failed every time. The label then said "Send a message to continue" again. > - Second, an exhausted system workspace repair stayed active and system-owned for hours. The board saw no owner, and a reply started nothing. > - This pull request lets a reply continue from the backup when the source sandbox is provably gone, and hands a spent system repair to the board with a named Retry. > - The benefit is that productive work always has a next action that can succeed, and nothing replays or runs twice. ## Linked Issues or Issue Description **What happened?** (a) A native warm task ran one successful turn. A later legacy follow-up retired the warm sandbox lease. The next user reply got a new sandbox (`created` lease). The executor found a verified backup but no reusable lease, so it threw `runner_harness_state_mismatch: backup_without_reusable_lease`. The run retried on the restart-recovery path, failed `native_remote_recovery_lease_mismatch`, and ended as `native_continuation_requires_reconciliation`. Workspace recovery then said "Saved work recovered. Send a message to continue." Each new reply repeated the same failure. (b) A run failed `native_workspace_sync_out_retry_exhausted`. System workspace recovery used its 6 bounded attempts and became `exhausted`. The recovery action stayed `active` with `ownerType: system`. The UI hides system-owned recovery actions, so the board saw no action. Nothing ran for more than two hours. **Expected behavior** (a) When Paperclip destroyed the source sandbox and no provider work followed the backup, the reply continues once from the backup. Ambiguous states still fail closed. (b) When the bounded budget is spent, the action names a board owner and a Retry action that can progress it. **Steps to reproduce** (a) Finish a warm native turn. Retire its reusable lease through obsolete-lease cleanup. Send a reply. The new run gets a `created` lease and fails `backup_without_reusable_lease`. (b) Let system workspace recovery reach its deadline or retry limit. Read the recovery action. It is still `ownerType: system`. Related merged work checked first: #19145, #19184, #19187, #19189, #19190, #18943 (added the guard for migrated sessions). Open recovery PRs (`gh pr list --search recovery --state open`) do not touch these paths. ## What Changed - `native-retired-backup-source.ts` (new): `retiredSourceAuthorizesNativeBackupRestore` decides whether a created lease may restore the backup. All must be true: the backup's source lease is `expired` or `failed`; it has a provider termination receipt with state `destroyed`; its `nativeHarnessBackup` stamp names this exact manifest SHA-256, session and runner instance; exactly one host lease maps to the provider allocation; and no provider durable event for the session was recorded after the backup completed. The provider event type set moved here and the executor imports it. - `native-session-executor.ts`: `shouldRestoreNativeHarnessBackupIntoSandbox` accepts `retiredSourceContinuation`. A created lease with a configured reusable lease evaluates it, beside the existing migrated-session path. The restore uses the existing verified restore and stamps the new lease. - `system-recovery.ts`: `updateSystemRecovery` moves an `exhausted` action to `ownerType: board` and writes a next action and workspace recovery message that name the attempt count, the last failure code, and **Retry recovery**. Retry (`requestWorkspaceRecovery`) already starts a new bounded generation and returns ownership to the system. - `doc/execution-semantics.md`: documents both rules. ## Verification - Red before the fix, green after: - `npx vitest run src/services/native-runtime/native-session-executor.test.ts -t "restores a warm backup into a created lease only when"`. Before: the destroyed-source case rejected with `runner_harness_state_mismatch: backup_without_reusable_lease`. After: it restores the runner state and provider home into the new sandbox and stamps the new lease. - `npx vitest run src/services/workspace-recovery.test.ts -t "hands an exhausted system export repair"`. Before: `exhausted` with `ownerType: "system"`. After: `ownerType: "board"`, the blocker says Retry recovery and never says "send a message", no provider run or lease resume happens, and Retry starts one new system generation. - Safety cases (fail closed): a stopped (not destroyed) source, a stamp for another checkpoint, provider work after the backup, a still-reusable `released` lease, a missing receipt, and two host leases for one allocation. These run in the executor test and in `native-retired-backup-source.integration.test.ts` against embedded Postgres. - Related suites pass: `workspace-recovery.test.ts`, `system-recovery-state.test.ts`, `native-session-executor.test.ts`, `task-retry.test.ts`, `explicit-native-continuation.test.ts`, `workspace-recovery-source.test.ts`, `native-workspace-export-retry.test.ts`, `native-remote-retry-admission.test.ts`, `native-workspace-warm-reuse.test.ts`, `native-workspace-sync-retention.test.ts`, `heartbeat-observed-workspace-preservation.test.ts`, `execution-control-work.test.ts` (2,224 tests). - `npx tsc --noEmit -p server/tsconfig.json` passes. - Not done: a live run against staging. Staging access was read-only. The staging evidence is the run event timeline described above. ## Risks - (a) A backup restore after a lost sandbox could rewind a session if newer provider state existed. The predicate requires a provider `destroyed` receipt and no provider event after the checkpoint. A stopped sandbox, a pending cleanup, or a still-reusable lease still fails closed with the old error. - (a) The event-time check can fail closed if a late event from the backup turn is written after the checkpoint time. That keeps the old behavior; it never authorizes a restore. - (b) The action's next action text stays as the exhausted message while a later Retry generation runs. The task blocker shows the live workspace recovery message first, so users see "Recovering saved work…" during that time. - Not changed: when the predicate fails closed, the run still follows the existing reconciliation path. That path can still end with the "Send a message to continue" receipt. That residual is narrower now and needs a separate change to the workspace recovery receipt. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic. Agentic coding with tool use (shell, file edits, read-only browser inspection of staging). Default context window. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
87dfba6818 |
fix(native): retry an unpublished accepted task reply durably instead of dropping it (#19200)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run commits its status decision and its final reply comment
in separate steps
> - Background presentation recovery publishes an accepted reply when
the live path does not record one
> - That worker dropped the run after one refused attempt and only threw
on an error, with no record on the run
> - A task could then show done while the user never saw the answer,
until an unrelated wake reloaded the queue
> - This pull request makes the publication of an accepted ordinary task
reply durable, bounded, and recorded
> - The benefit is that a task cannot close silently without its answer
## Linked Issues or Issue Description
Refs #19130 (verify requested publication at finalization, for
documents), #19183 (deliver messages saved while a native run
finalizes), #19186 (stale owner done). This change does not duplicate
them. It covers the final reply comment.
**What happened?**
A native run accepted a result with disposition `done`. The status
decision set the task to done. The accepted final reply was not
published as a task comment. The run had no publication record (no
`run.presentation.resolved` event and no `presentationDecision`). The
reply was published by background recovery 58 minutes later, after an
unrelated wake reloaded the presentation queue.
**Expected behavior**
The accepted reply is published promptly. A refused or failed attempt is
retried with backoff and never posts twice. If publication fails
permanently, the run and the task show a named failure with an owner.
**Steps to reproduce**
1. Commit an accepted native task result with disposition `done`, with
the final reply not yet visible to presentation recovery.
2. Run the native response presentation worker check once. The attempt
is refused.
3. On `master`, the worker returns `null` and drops the run. Nothing
records the attempt. Nothing retries it until another notification for
that scope.
**Paperclip version or commit**
`master` at
|
||
|
Devin FoleyandPaperclip
|
07a2941f0c |
fix(recovery): reclaim verified inactive provider caches on full disks (#19192)
Recover from a full-disk workspace save by reclaiming only a verified inactive provider installation under the existing source and maintenance authority. Commit the preparation inventory in Core before changing cache contents. Bind interrupted cleanup to stable filesystem identities and keep small staged generation markers when directory birth times are unavailable. Skip read-only or permission-denied assets and stop freeing blocks once marker space is available. Preserve task files, Git, session history and uncertain action receipts. Only the normal saved-workspace and provider-stop receipts permit continuation. Validated with 446 complete recovery regressions on the repair revision, subsequent focused recovery and transport suites, typecheck/build, and all exact-head hosted CI. Greptile reviewed the final head at 5/5 with no unresolved findings. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
245b4d3b79 |
Preserve cancelled workspaces after controllerless Stop (#18820)
Admit cancelled workspace preservation after an audited Stop acknowledgement when no live controller received an interrupt. Retain the strict dispatched receipt for provider termination and operator handoff. Existing source, lease, coordinator and result ownership checks remain enforced. Validated with complete Stop/preservation suites (134 tests), real-file recovery, workspace typecheck and build, and green hosted CI. Greptile and Paperclip both reviewed the exact head at 5/5 with no unresolved findings. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
3b17b5ce15 |
fix(native): keep a reassigned-away run from closing the new owner's task (#19186)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native agent run reports "done" in an accepted result. A finalizer then decides the task status. > - The board can reassign a task while the run is in flight. The old run is cancelled, but its accepted result still goes to the finalizer. > - The finalizer did not check who owns the task now. The old owner's "done" closed the task of the new owner. > - The new owner never did the work, and the task shows done with no reply from the current owner. > - This pull request makes a done report apply only when the reporting run's agent still owns the task. > - The benefit is that reassignment keeps its meaning: the new owner keeps the open task, and the old result stays as history. ## Linked Issues or Issue Description **What happened?** On a hosted staging instance, agent A started a native run on a task. Agent A finished the work and submitted a "done" result. Then the board reassigned the task to agent B, and Paperclip cancelled agent A's run with `issue_reassigned`. About 45 seconds later, the native finalizer committed agent A's done decision with reason `completion_claim_policy_accepted`. The task became done. Agent B's runs were then cancelled because the task was terminal. Agent B never worked on the task. **Expected behavior** A done report from a run that no longer owns the task must not close the task. The accepted result stays as history. The task status stays as it is for the current owner. **Steps to reproduce** 1. Assign a low-risk native task to agent A and start a run. 2. Submit a done result for the run. 3. Reassign the task to agent B before finalization. 4. Finalize the run. Before this change, the task becomes done. **Paperclip version or commit** Observed on staging at `b9fbbf8d30`. Reproduced on `master` at `06826915d7`. Related: #19130 adds a publication check for requested documents at finalization. It does not check task ownership. ## What Changed - `status-arbiter.ts`: add `ownsNativeTaskCompletion` and `supersedeStaleOwnerCompletion`. A done decision from an agent that is not the current assignee (another agent, a user, or nobody) becomes `preserve` with reason `completion_owner_superseded` and no effects. Other decisions do not change. This is the same ownership rule that the provider-failure path already uses (`ownsProviderFailureDecision`). - `native-run-finalizer.ts`: apply the rule after arbitration and before the deliverable check. Conversation tasks keep their existing path. - `native-status-reassessment.ts`: apply the same rule, so new evidence from a reassigned-away run cannot close the task later. - The status committer does not change. A reassignment during finalization already increments `statusVersion` (migration 0358). The committer then replans, and the retry applies the new rule. - `doc/execution-semantics.md`: describe the rule next to the existing "a late finalizer cannot change a reassigned task" statement. ## Verification New file `server/src/services/native-runtime/native-stale-owner-completion.test.ts`. It uses embedded PostgreSQL, the real coordinator store and the real finalizer. - Control: the owning run's done result closes the task with `completion_claim_policy_accepted`. - Reassigned before finalization: the task stays `in_progress` with agent B. The decision is `completion_owner_superseded`. The accepted result row is unchanged. - Reassigned after arbitration and before the status commit: the commit replans and the task stays `in_progress`. - Reassessment control: later approved evidence closes the owner's task with `decision_superseded_by_new_evidence`. - Reassessment after reassignment: later approved evidence from the old run does not close the task. The decision is `completion_owner_superseded`. This case fails when the reassessment rule is disabled. - Pure policy cases: another agent, a board user and no assignee all preserve status. A non-done decision passes through. Red before the fix (same tests, fix not applied): ```sh cd server && npx vitest run src/services/native-runtime/native-stale-owner-completion.test.ts # Tests 2 failed | 1 passed — both reassignment cases: expected "in_progress", received "done" ``` Green after the fix: ```sh cd server && npx vitest run src/services/native-runtime/native-stale-owner-completion.test.ts # Tests 10 passed cd server && npx vitest run src/services/native-runtime/paperclip-control-plane-port.test.ts \ src/services/native-runtime/native-completion-feedback.test.ts src/services/native-runtime/native-deliverable-feedback.test.ts \ src/services/native-runtime/native-skill-test-status.test.ts src/services/native-runtime/native-routine-status.test.ts \ src/__tests__/native-status-arbiter-corpus.test.ts src/__tests__/native-status-reassessment-work.test.ts \ src/services/native-runtime/agent-chat-response-recovery.test.ts src/__tests__/native-run-projection-work.test.ts \ src/services/native-runtime/task-reassignment.test.ts src/services/native-runtime/native-provider-failure.integration.test.ts # 11 files, 614 tests passed ``` I did not run the full `pnpm test:run`, `pnpm -r typecheck` or `pnpm build` locally. CI runs them. ## Risks - A native run that reassigns its own task to a user or to another agent, and then reports done, no longer closes the task through arbitration. The status that the agent set through the API stays. This matches the existing provider-failure ownership rule. - A native run on a task with no assignee cannot close it through arbitration. Native task runs start from an assignment, so this is not expected in normal use. - Review runs and conversation tasks keep their existing paths. Review outcomes already preserve status before this rule. - No schema change and no migration. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Provider: Anthropic. Model: Claude Opus 5.5 (`claude-opus-5-5`), in the Claude Code agent harness. - Capabilities used: tool use, shell and code execution, repository search, browser reads of staging, and extended reasoning. - The model wrote the change, the tests and this description. A human requested the fix. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
2141efc097 |
fix(native): answer Stop during slow native startup with 409 and settle it as cancelled (#19189)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Operators stop a running agent run with `POST
/api/heartbeat-runs/:runId/cancel`
> - For a native run, Stop waits up to 30 seconds for the native session
to open before it can deliver the cancel
> - A remote native session can take longer than 30 seconds to open.
Then the wait fails with `native_cancellation_pending_recovery`, and the
route answers HTTP 500
> - The Stop intent was already saved. The startup owner fenced the late
session in memory only, so the run ended as a provider failure, not as a
cancellation
> - This pull request maps that outcome to a clear 409 and makes the
late session acknowledge the saved Stop intent
> - The benefit is a consistent cancel result and a run that ends as
`cancelled` when an operator stops it during a slow startup
## Linked Issues or Issue Description
**What happened?**
An operator stopped a native run about 40 seconds after it started. The
run was still opening its remote provider session (session startup took
about 105 seconds). The cancel request waited 30 seconds for the startup
handle, then returned HTTP 500 `{"error":"Internal server error"}`. The
run record showed `executionCancellation.state = "requested"` and
`nativeCancellation.dispatchState = "pending"`. When the session opened,
the executor fenced it and threw `native_finalization_missing: session
returned no semantic result`. The run ended `failed` with
`native_session_interrupted`, and the pending Stop intent was never
acknowledged.
**Expected behavior**
The cancel route returns a clear result and never an internal error. If
the Stop is saved but the session is still starting, the route returns
409 with code `native_stop_pending`. When the late session opens, it
acknowledges the saved Stop, and the run settles as an operator
cancellation.
**Steps to reproduce**
1. Start a native run whose provider session takes more than 30 seconds
to open.
2. Stop the run from the board while the session is still opening.
3. See HTTP 500 from the cancel route, and see the run end `failed`
instead of `cancelled`.
## What Changed
- `server/src/services/heartbeat/run-control.ts`: the cancel path
catches `native_cancellation_pending_recovery` from native dispatch. It
throws `409 native_stop_pending` instead. The run stays `running` and
keeps its saved Stop intent. No terminal transition or release happens.
- `server/src/services/native-runtime/native-session-executor.ts`: when
a late startup handle opens after a Stop caller timed out, the executor
reads the run. If a run-scope Stop intent is still `pending`, it calls
the durable `cancelNativeSession(..., { db, scope: "run" })` against the
published handle. This records the audited acknowledgement (`dispatched:
true`). The existing in-memory fence still runs. If the acknowledgement
fails, the executor logs a warning and keeps the previous behavior.
- New regression
`server/src/services/heartbeat/native-stop-pending.test.ts` (real
Postgres): startup-wait timeout and remote-ack timeout both give 409
`native_stop_pending`. The run stays `running` with its Stop intent.
Other native Stop errors stay unchanged.
- New regression in `native-session-executor.test.ts`: a Stop that times
out on startup gets its saved intent acknowledged when the late handle
opens. The provider session is cancelled once, and no turn is submitted.
## Verification
- Red before the fix: `native-stop-pending.test.ts` failed (raw
`NativeCancellationPendingRecoveryError` / `Error`, not `HttpError`).
The executor regression failed (no acknowledgement write).
- Green after the fix: `cd server && npx vitest run
src/services/heartbeat/native-stop-pending.test.ts`
- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t "late
startup"`
- Related suites:
`src/services/native-runtime/native-session-executor.test.ts`,
`src/services/heartbeat/`,
`src/services/native-runtime/native-stop-accounting.integration.test.ts`,
`src/__tests__/agent-live-run-routes.test.ts`.
- Evidence: the incident run had cancellation recorded at 01:12:01Z and
the 500 at 01:12:32Z. This matches the 30-second
`waitForNativeSessionStartup` deadline. Its final error matched the
startup-stop branch in the executor.
## Risks
- Low to medium. The 409 only replaces an error that was already a
failure. The caller still sees "not stopped yet", and the run is not
marked cancelled.
- The late acknowledgement runs only when a run-scope intent is
`pending` and the handle is published. It uses the same durable cancel
path that the reattach branch already uses. It can commit the normal
Stop status decision for the issue. That is the same effect a timely
Stop has.
- If the acknowledgement fails, the code falls back to the previous
in-memory fence. A Stop never gives a second executor: no provider turn
is submitted after the handle opens.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandClaude Opus 5.5
|
f1516e8009 |
fix(native): stop a session.open model rejection instead of a doomed bootstrap retry (#19190)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native Runner runs start a provider session with the `session.open` command before any prompt > - A definite `session.open` rejection is treated as a pre-session failure and gets a bounded same-run bootstrap retry > - The runner also reports a closed classification. Two classifications say that the provider cannot run the selected model: `requested_model_unsupported` and `grok_model_mismatch` > - Paperclip ignored that classification. It scheduled a retry that must fail the same way. On a remote sandbox with a retained runner PID, the due retry entered restart recovery. Restart recovery has no checkpoint for a bootstrap retry, so it failed closed as `native_restart_recovery_blocked` / `ambiguous_remote_recovery_evidence` > - The task then stayed `blocked`. The recovery actions settled as no-replay holds, so no active action told anyone to change the model, and the parent task stalled too > - This pull request classifies those two definite rejections as the existing `native_provider_model_rejected` failure > - The benefit is a terminal failure on the first attempt, an accurate error, and an active board recovery action that names the next step: choose a supported model ## Linked Issues or Issue Description **What happened?** On a hosted staging instance, a Grok agent run failed at `session.open`. The runner log said `ACPX sidecar command session.open was rejected (retryable=false, classification=grok_model_mismatch)`. Paperclip recorded `recoveryMode: "bootstrap_retry"` and scheduled a same-run retry 30 seconds later. The run had a remote sandbox and a recorded sandbox PID, so the due retry went to restart recovery. Restart recovery found no checkpoint and no provider events and failed closed with `ambiguous_remote_recovery_evidence`. The run ended as `native_restart_recovery_blocked`. The task stayed `blocked` with no active recovery action for more than one hour. Its parent task stayed blocked on it. **What did you expect to happen?** The selected model cannot start, so a retry cannot succeed. The run should fail once with the model-rejection error. The task should show the existing "choose a model" action, owned by the board, until someone changes the model and retries. **Steps to reproduce** 1. Start a native run whose `session.open` command fails with `classification=grok_model_mismatch` (or `requested_model_unsupported`). 2. Read the `native_run_finalizations` row. Before this change it is `retryable_failure` with `recoveryMode: "bootstrap_retry"` and a `nextAttemptAt`. 3. On a remote sandbox run with a recorded PID, the due retry enters restart recovery and fails as `native_restart_recovery_blocked`. **Paperclip version or commit** `master` at `f5f5184f23`. Related: #18910 routes due remote retries with a retained PID to restart recovery. #18908 continues verified failed native preparation. Open PRs #19177, #19180, #19182 and #19184 change no-replay hold settlement and workspace hold release. They do not change this classification. This PR does not duplicate them. ## What Changed - `nativeSessionFailureSourceCode` (`server/src/services/native-runtime/native-session-executor.ts`) returns `native_provider_model_rejected` for a definite `session.open` failure or rejection whose runner classification is `requested_model_unsupported` or `grok_model_mismatch`. Other definite `session.open` rejections still return `provider_session_open_rejected` and keep the bounded startup retry. An indeterminate `session.open` outcome stays `native_session_interrupted`. - `native_provider_model_rejected` is already a permanent failure. The run stops on the first attempt with no `nextAttemptAt`, so it never enters the restart-recovery retry path. - When no provider session exists, the coordinator `nextAction` and the board recovery action `nextAction` say: choose a supported model for this agent, then retry the task. - The existing product paths for `native_provider_model_rejected` apply: the task presentation shows `choose_model`, and the recovery service treats it as `configuration_incomplete`. - `doc/execution-semantics.md` describes the rule under "Native provider model capacity". ## Verification Red before the fix (new test, unchanged production code): ```sh cd server && npx vitest run src/services/native-runtime/native-startup-retry.integration.test.ts # Tests 2 failed | 21 passed # failureCode: native_session_interrupted, phase: retryable_failure, recoveryMode: bootstrap_retry, # nextAction "Retry provider bootstrap on this same run ..." - the same record seen on staging ``` Green after the fix: ```sh cd server && npx vitest run src/services/native-runtime/native-startup-retry.integration.test.ts # Tests 23 passed cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts \ src/services/native-runtime/native-startup-retry.integration.test.ts src/services/task-interruption-presentation.test.ts # Test Files 3 passed, Tests 1186 passed pnpm --filter @paperclipai/server typecheck # exit 0 ``` The new integration cases use embedded PostgreSQL and the real executor failure path. For both classifications they check: - coordinator `terminal_failure`, `failureCode: native_provider_model_rejected`, `nextAttemptAt: null`, attempt 1 - run `failed` with `errorCode: native_provider_model_rejected` - task `blocked`, execution lock released - an `active` board recovery action with cause `native_provider_model_rejected`, return owner set to the agent, and the choose-model next action - `settleUnrecoverableExecutions` leaves the action `active`, so it does not become a silent no-replay hold - `claimNativeSessionResumptions` claims nothing, so there is no second executor The unit test also checks that `session_ensure_failed` keeps its bounded retry and that an indeterminate model rejection stays unclassified. A live Grok run with a mismatched model was not run. The failure record in the test matches the staging record field by field. ## Risks - Low risk. The change adds two closed runner classifications to an existing permanent failure code. Other `session.open` rejections keep their current retry. - A model mismatch that a retry could fix would now need an explicit retry. The runner already marks these rejections `retryable=false`. - This PR does not change restart recovery. A remote bootstrap retry with a retained sandbox PID for other `session.open` causes can still fail closed as `ambiguous_remote_recovery_evidence`. That needs a separate change with remote stop evidence. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used in Claude Code with tool use (shell, file edits, browser reads of the staging API). It did the investigation, code change, tests and PR text. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
f7a4c5678f |
feat(runtime): pre-copy completed harness history before enrollment (#19137)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - An ephemeral web server needs provider history in shared object
storage.
> - Offline enrollment must preserve old backups before it can detach a
volume.
> - A large copied dataset still exceeded the maintenance deadline with
parallel transfers.
> - Most of its bytes belong to successful turns with released runtime
authority.
> - This pull request pre-copies that history while unrelated executions
continue.
> - The final offline migration still verifies every source and grants
enrollment authority.
## Linked Issues or Issue Description
Refs #19108, #19038, #18943.
**What happened?**
A copied instance has 1,953 local harness backups and 54.8 GB of
declared source bytes. A resumed parallel migration still reached its
25-minute deadline. It cannot safely enroll by moving all history during
maintenance.
**Expected behavior**
Copy eligible completed history while the server remains online. Defer
active, failed or changed sources. Keep all existing data and final
enrollment checks.
**Steps to reproduce**
1. Retain many local native harness backups on a volume-backed instance.
2. Attempt a full offline conversion within the maintenance deadline.
3. Observe that most time moves historical bytes rather than current
execution state.
## What Changed
- Add finite `native-harness-prestage.mjs` preparation from a normal
runtime build. Require explicit opt-in and shared object storage.
Process one source at a time with a 90-minute deadline.
- Use the recorded released runtime owner of each successful, finished
turn. Do not assert that all writers stopped.
- Recheck checkout, related queued/running work, successful result
ownership, session, Runner, exact lease stamp and unchanged backup at
database publication fences.
- Defer source changes, malformed source JSON and unsafe backup roots.
Log each such source and its reason. Keep database failures and
cancellation fatal. Keep completed objects valid for retry. Never return
an enrollment-ready receipt.
- Reuse the bounded migration source walker. Keep offline admission and
final verification unchanged.
- Keep private lifecycle state, task ownership and provider policies
unchanged. Document the operator path and its limits.
## Verification
- A zero-source scan now checks the existing database obligation query.
A known mismatch blocks with a source-missing receipt; empty
fresh/already-durable scans and nonempty deferred scans remain valid
advisory results. The prior command fails the mismatch regression. All
16 command and runtime-bundle tests pass for this follow-up.
- At head `7ba3dbc7aa`, 87 focused command, database-timeout, storage,
archive, publication and runtime-bundle tests passed with real
PostgreSQL and filesystem restoration.
- The current follow-up passes all 68 backup and archive tests. A source
renamed after preparation fails the new regression on the previous head,
then defers and permits the next backup with this fix. An archive output
ENOENT remains fatal. Existing database-error and cancellation tests
pass.
- New race tests fail against the previous storage owner: a changed
backup, queued turn or revoked lease otherwise publishes.
- Tests prove unrelated executions stay running, same-task work defers,
failed history stays deferred, retry verifies existing objects, and
private bytes restore after deleting the old home.
- Command review fixes require the configured existing vault key, retain
it for the process, set database lock/statement/idle bounds, close
clients on cancellation, and cap shutdown at ten seconds. Abort
immediately sets failure status and prints one blocked receipt. A real
child with real shortened timers reproduces the old missing-receipt exit
and verifies the fix. Tests also use a real PostgreSQL lock.
- Malformed manifest or runner JSON and symlink replacement fail the new
preparation and publication-fence regressions on the previous code. The
fixed command defers each damaged source and copies the next eligible
backup. Separate tests keep database SyntaxError and cancellation fatal.
Offline migration still rejects damaged sources.
- The bundle test runs the emitted preparation entry with plain Node
without starting the app.
- A prior copied-data experiment completed with 1,738 new publications,
nine verified current objects and 206 deferred sources. Later checks
found different database and filesystem capture times: 43 source runs
were absent from the older database snapshot, and one required
historical manifest was absent from the retained filesystem archive.
This experiment does not prove a consistent restore or enrollment
readiness.
- Full current-head CI at
|
||
|
DottaandClaude Opus 5.5
|
a994a648f1 |
fix(workspaces): name copy-back failures on the run and end failed save operations (#19187)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native run in a Daytona sandbox saves its workspace after the provider turn. This save is "copy-back". > - On staging, about 20 runs kept a saved result, but copy-back failed three times. The issues became blocked with `native_workspace_sync_out_retry_exhausted`. > - #19152 added a redacted cause for a sandbox command that exits with an error. Three gaps stayed open: a provider that refuses the command still gives a bare code, the run does not show the cause, and the failed save operation stays "running". > - The first failure also had the wrong name. The runner kept the result but did not prove a durable suspension. The executor reports this as `native_session_lease_lost`, and the workspace operation shows only that text. > - This pull request names each of these failures with closed, redacted labels. It also ends a failed save operation as "failed". > - The benefit is that operators can see why copy-back failed and which save is still active. Retries and saved results do not change. ## Linked Issues or Issue Description Refs #19152, #19145. These are the earlier copy-back diagnostics and retry-scheduling repairs. This PR does not duplicate them. **What happened?** A native task ran in a Daytona sandbox and its result was accepted. The run then failed with `native_workspace_sync_out_retry_exhausted`. The run had no error text. The automatic copy-back retries each failed in under one second, with only `workspace_sync_out_failed`. The final `remote_workspace_snapshot` operation stayed "running" after the retries stopped. The first `workspace_finalize` operation showed only `native_session_lease_lost`. The run log showed that the runner shutdown timed out before durable suspension. The lease was not lost. **Steps to reproduce** 1. Run a native task in a Daytona sandbox until its result is accepted. 2. Make the sandbox provider refuse the copy-back command (for example, a lease that is no longer active), so that no exit status comes back. 3. Let the automatic retries stop. Then read the run and its workspace operations. **Expected behavior** The run, the coordinator, and the failed operations name a redacted cause. The failed save operation is "failed" with a finish time. **Actual behavior** The run has no cause. The save operation stays "running". A provider refusal keeps only the bare `workspace_sync_out_failed` code. ## What Changed - `remote-workspace-transport.ts`: the provider can throw instead of returning an exit status for the copy-back helper, the capture, or the archive download. `remoteWorkspaceFailureCause` then returns `remote_workspace_operation_failed: operation=<op> transport=<label>[ code=<ERRNO>]`. The label is from a closed list (for example `sandbox_lease_inactive`, `permission_denied`, `sandbox_not_running`, `transport_timeout`, `transport_closed`, `daytona_sandbox_not_found`, `provider_error`). Provider text is never copied into the cause. The thrown error does not change, so preparation errors, `classifyNativeWorkspaceFailure`, and other callers keep their contracts. Archive-safety and Git-metadata refusals get no label, because they have their own diagnostics. - `remote-workspaces.ts`: when the last active invocation of a save fails, the save operation becomes "failed" with `exitCode`, `finishedAt`, a redacted `stderrExcerpt`, and `metadata.lastFailure`. A retry of the same boundary sets it to "running" again. A success sets it to "succeeded". After the final save commits, a failed save is retired for export cleanup in the same way as a running save. - `remote-workspace-snapshot-retention.ts`: a failed save that is not retired keeps its snapshot pinned, as it did while it was "running". Its retry can still use an uploaded snapshot. - `native-run-finalizer.ts` and `native-workspace-finalizer.ts`: the redacted cause of each copy-back attempt goes to the coordinator `failureDetail.failureCause` and the run `resultJson.failureCause`. When retries stop, the run error adds `Last cause: <cause>`. The server checks the cause against a fixed grammar again before it writes it. A cause with a path or other text is dropped. - `native-post-result-failure.ts` (new), `native-session-executor.ts`, `runtime-execution.ts`: a failure after an accepted result gets the label `runner_settlement_incomplete` when the runner did not prove a durable suspension. The failed `workspace_finalize` operation records this closed diagnostic as `failureDiagnostic`. The coordinator records it as `native_post_result_failure: category=<label>`. The control code `native_session_lease_lost` and its fencing do not change. ## Verification - Red, before the fix (the new tests run against `master` source): - 4 provider-refusal cases and the archive download failure: no redacted cause was available, so the copy-back recorded only the bare code. - Failed save, with only the transport fix applied: `- "status": "failed"` / `+ "status": "running"`. - Run cause after retries stop: `expected null to be 'remote_workspace_operation_failed: …'`. - Green, after the fix: `cd server && npx vitest run src/services/remote-workspace-transport.test.ts src/services/remote-workspace-save-concurrency.test.ts src/__tests__/native-workspace-finalization-work.test.ts src/services/native-runtime/native-post-result-failure.test.ts src/services/remote-workspaces.integration.test.ts src/services/remote-workspace-snapshot-retention.test.ts` passes. - Safety cases: - The failed save is retried on the same boundary operation, records "succeeded", and keeps the saved file bytes. - `daytona_sandbox_not_found` is still classified as `workspace_sync_out_unrecoverable`. - An archive-safety refusal keeps its exact text. - A forged cause with a path never reaches the run or the coordinator. - A run-level timeout cause (`remote_workspace_operation_timed_out: capture (configured timeout 300000ms)`) reaches the run and the coordinator. - `heartbeat-native-remote-failure-retention.test.ts` passes: a preparation failure keeps its exact run error. - The existing integration tests for retired saves, poisoned reservations, duplicate invocations, and snapshot retention pass. They prove export cleanup and identity protection. - `tsc --noEmit` for `server` reports no errors. - Not done: a live Daytona copy-back. This PR adds diagnostics and corrects the operation state. It does not change when a retry runs. ## Risks - Thrown copy-back errors do not change. Only the recorded cause is new. The `daytona_sandbox_not_found` and archive-safety classifications are tested. - A failed save operation is now "failed" before the final save commits. Retirement, export cleanup, and snapshot pinning now also select these failed rows. They select only rows with `lastFailure` and no retirement receipt. - Privacy: each new cause is a closed label or a server-built code. The server checks it again with an anchored grammar before it writes the run. No sandbox output, path, or message text is stored. - The retry count, the retry delay, and the classification of each failure do not change. No provider turn is added. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Provider: Anthropic. Model: Claude Opus 5.5 (`claude-opus-5-5`), in the Claude Code agent harness. - Capabilities used: tool use, shell and code execution, repository search, read-only browser reads of staging, and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Nicky Leach
|
8c062fa481 | fix(native): deliver plugin VM Runner usage to Core (#19162) | ||
|
DottaandClaude Opus 5.5
|
512fbea0bb |
fix(native): deliver messages saved while a native run finalizes (#19183)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A user can reply on a task while the agent's run is still finishing > - A remote native run first commits its result and its status decision (for example `done`). Then its sandbox environment settles (harness backup, warm retention). > - A message that arrives after the final turn took its input is saved as a deferred receipt behind that run > - The saved-message recovery path holds the receipt while the source run still owns its environment lease or its workspace copy-back owner > - Nothing notified that path when the lease or the owner settled, so the messages waited for an unrelated company event > - This pull request notifies the existing deferred-receipt work when a run's environment lease settles or its workspace finalization owner is released > - The benefit is that messages sent while a run finalizes are delivered once, soon after the run settles, and a task closed in that window is reopened for them ## Linked Issues or Issue Description **What happened?** A user posted three plain comments on a task while the agent's remote native run was finalizing. The final turn had already called its finish tool. The comments were saved in one deferred receipt. The run then committed `done` and its status became `succeeded`. No follow-up run started. The task stayed `done` with three unanswered requests. About 20 minutes later, an unrelated event in the same company woke the deferred-receipt worker, and the messages were delivered. **Expected behavior** A message that arrives after the last turn took its input, but before the run releases, gets one follow-up turn as soon as the run settles. If finalization marked the task `done` in that window, the task is reopened for the message. Messages that the finished turn already consumed are not replayed. **Steps to reproduce** 1. Start a native run with a remote (sandbox) environment. 2. After the agent calls its finish tool, post a user comment on the task. 3. Let finalization commit `done` and project the run as `succeeded`. 4. Observe that the comment's deferred receipt stays `deferred_issue_execution` after the environment lease settles. **Root cause** A remote native run is finished by native finalization, which projects the terminal status. The in-process run completion (`releaseIssueExecutionAndPromote`) does not run for it. Its saved messages depend on the stranded-queue sweep in `resumeDeferredExecutionReceipts`, which calls the release with `resumeStrandedQueue`. That release correctly waits while `hasPendingRunCleanup` is true (an unsettled environment lease or a workspace finalization owner). Finalization notifies the deferred-receipt worker when it projects the run, but the cleanup is still pending at that time. The worker then drops the entry. Lease settlement and owner release did not notify the worker again. Refs #19122 (saves a redirect sent while a stopped run settles; this change reuses the same deferred-receipt delivery path for normally completing runs). ## What Changed - `environmentService.releaseLease` notifies deferred-execution work for the company when a run's lease leaves `active` or `pending_cleanup` (released, retained, or failed). - `environmentService.releaseLeasesForRun` notifies deferred-execution work in the same way. - `withNativeWorkspaceFinalizationOwnership` notifies deferred-execution work when it releases its owner. - Delivery uses the existing path: `resumeDeferredExecutionReceipts` → stranded-queue release → deferred wake promotion (with the existing reopen rule for user messages). No new delivery mechanism. - New regression in `deferred-execution-receipt-work.test.ts` with the real deferred-receipt worker, the real release adapter, and the real lease service. - `doc/execution-semantics.md`: one additive paragraph for messages sent while a run finalizes. ## Verification - Staging evidence (read-only): the task's saved queue held the three comments as one `deferred_issue_execution` receipt after the run succeeded. The run had no `run succeeded` lifecycle event (it was finished by native finalization, not run completion). The receipt was promoted about 20 minutes later with `reopenedFrom: done` by the same recovery path. - Red before the fix: the new test failed with `expected null to be truthy` (the receipt stayed deferred with no run after the lease settled). - Green after the fix: `cd server && npx vitest run src/__tests__/deferred-execution-receipt-work.test.ts -t "finishing native run"`. - Safety cases in the new test: - The receipt stays deferred while the environment lease is active (cleanup holds still apply). - After settlement, exactly one run carries both messages in order, and the done task is reopened once. - A second settlement signal does not deliver the messages again. - Related suites: `deferred-execution-receipt-work`, `environment-service`, `environment-cleanup-work`, `native-environment-settlement-work`, `idle-sleep-safety`. - Not done: a live staging reproduction at the fixed revision. Deployment is out of scope. ## Risks - The deferred-receipt worker reloads the company's deferred receipts after each run lease settlement. Each reload is bounded and runs only when deferred receipts exist. The notification is transactional, so a rolled-back release does not wake the worker. - Low risk: no schema change, no API change, and no change to admission or reopen rules. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file edits, browser for read-only staging diagnostics). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
a0250284e3 |
fix: preserve proposal revision turns and prove chat completion (#19105)
## Thinking Path > - Paperclip manages work done by AI agents. > - A completed delegated task must report its result to the originating chat. > - That report uses a real provider turn and must stay in cost accounting. > - Ordinary first-task proposals must also support the promised chat acceptance path. > - A revision response can reject an older proposal before it creates the replacement. > - The old rejection transition clears that active response's ownership and blocks its next tool. > - This change proves required completion work, guides ordinary proposal policy and preserves the revision turn under the existing locks. ## Linked Issues or Issue Description No separate issue exists. The original failures remain in [the complete Product run](https://github.com/paperclipai/paperclip/actions/runs/41915965118). The independent continuation repair is #19076. Related #12209 concerns prior-run confirmation binding. This change fixes the ownership reset during rejection, rather than changing binding authorization. **What happened?** The complete 20-case run finished with 13 passes and 7 failures. Three delegated-chat cases counted one required `chat_task_completed` provider turn as unexpected work. Two first-task agents chose optional human-only cards, then correctly received 403 when they tried conversational acceptance. A [reviewed two-case candidate run](https://github.com/paperclipai/paperclip/actions/runs/41931445311) then used the changed first-task skill. Both models inherited the ordinary resolver policy. Clarify/propose/accept passed all 12 predicates. Revise/accept failed after the new response rejected its predecessor's proposal. The saved rejection was followed by `paperclip_runner_tool_binding_not_authorized` on its next tools. The final `native_session_lease_lost` label followed those binding failures; this PR does not assume lease expiry caused them. Both cells cleaned up and published their original grades. **Expected behavior** Keep the existing ordinary-turn bounds and account for one independently proved completion notification. Inherit the normal resolver policy for ordinary proposals that accept chat replies. When an active response rejects an older proposal, save the rejection and retain that response's current ownership so it can create the revised proposal. Keep human, current-source and native completion-review rejection behavior and all real approval restrictions. **Steps to reproduce** The original full run used `4740d251`. The two-case candidate used trusted workflow `7d1e9b34` and target `74da56c4`. Run the focused provider-free controls below. The dual-run API regression first fails because rejection changes the issue to `todo` and clears the newer run's execution and checkout fields. ## What Changed - Prove one completion notification only for native Codex/local `multi-repository`, `clarify-reuse` and `plan-handoff`. Keep the ordinary bounds of 2, 3 and 4. Preserve actual totals of 3, 4 and 5, all run IDs and all usage. - Read those declared totals from the current catalog, which now includes the completion reply. Check the total once and prove that exactly one turn is the required notification. Do not add another turn while waiting. The required-server controls use the actual catalog tasks. - Bind the notification to the same company, agent, conversation generation, completed child, single delivery, accounted native run, saved reply and rendered result link. Reject extra work, duplicate deliveries, unknown events and mutations, including native work without a Paperclip tool call. - Collect workspace operations through the real run API for these three proofs. Keep other collector calls unchanged. Keep the proof controls in the existing required server test lane. - Stop idle polling on a permanent completion-proof error. Continue to retry transient observations within the original 240-second window. - Guide ordinary chat-accepting first-task proposals to omit optional `resolverPolicy` and inherit server/company defaults. Preserve explicit and governed human-only restrictions. Do not lower or recreate saved restricted cards. - Under the existing locked issue and interaction rows, preserve the active responding run when it rejects a different source run's proposal. Change only the ordinary requeue predicate. Keep the saved rejection, notifications, activity and existing Telemetry calls. Keep global status, lease and tool-authority guards unchanged. - Extend the existing native API controls with the actual dual-run rejection/revision sequence and policy, provenance, human, current-source and native completion-review countercontrols. No new workflow or test framework is needed. ## Verification - Current combined head is `a8567ded67e5ab811675390e9a5c5ed2194aee2f`, integrated cleanly on `fb96537e28c4c7bfced4939909cb334392320868`. The upstream completion-reply waiter is preserved. [Current required CI](https://github.com/paperclipai/paperclip/actions/runs/41940780852) passes. Actual test checkout `ff93c034d7bb40f3df7bc75cf51828fd07bb0d29` merges this head into `9c6993f0aab7ebb22a26d78f5d227accb35a6ad1`. - Fresh Greptile is 5/5 on exact head `a8567ded`, with no blocking finding. All addressed review threads are resolved. The separate automated reviewer emitted a paused-work response; that response is not a review approval. - Current required server CI actually passed all 11 native confirmation API cases, 46 completion-proof controls and 13 onboarding asset checks. Canonical typecheck, Build, the native package/server suites, all browser and serialized shards, `ci / e2e` and `ci / verify` pass. No current-head CI retry was needed. - Local dual-run RED: the unmodified transition returned HTTP 200 and saved rejection, then changed the issue to `todo` and cleared the newer run's execution/checkout fields. Whole setup and check took 28.95 seconds; the case took 596 ms. - Local treatment: all 11 real native/API cases pass. The same-session response retains execution/checkout ownership, reads task context and persists the recorded-shape revised confirmation. Its stored lease receipt stays intact and one rejection activity is saved. Human-only rejection, current-source rejection, human rejection with/without a newer owner, Plan/Ask, foreign provenance, authentication and formal approval controls pass. - All 59 existing confirmation resolver controls pass in 16.62 seconds. These cover cancellation, lost ownership, company/user scope, policy changes, transactional rollback, concurrent retries and stale revisions. - All 46 completion controls pass in 4.07 seconds. The two added clock controls preserve the original idle deadline and prove immediate permanent-proof failure plus transient observation retry. The existing live collector-to-proof and native mutation negatives remain in this required server file. - The first current-catalog integration check failed 5 controls in 2.70 seconds. The old consumer added the already-declared reply again and could accept an extra ordinary turn. The corrected consumer passes all 46 controls using the actual catalog. The existing full 20-case catalog/reporting compatibility check also passes in 4.93 seconds. Case identities, configured models and reporting match the declared 20. - The API controls used one owned loopback PostgreSQL instance through existing supported test hooks. It was stopped and removed. Earlier missing-library/setup failures and one corrected current-source test expectation remain retained; they are not counted as product failures or passes. - The unchanged first-task skill has SHA-256 `b789090996e7925676590c006418eb8b57f4514e646f6bca18ffd7b2b8e864de`. Both candidate runtime copies matched it. The inherited policy was observed in both models; one complete acceptance journey passed. This does not qualify the new rejection repair. - The three retained real completion streams pass the stronger local proof, including native items, workspace observations and recorded DOM links. Their original failed grades stay unchanged. - Earlier related checks passed 153 hiring/non-execution controls and 13 asset checks. Their source is unchanged. Required discovery includes the completion, API and asset files exactly once. - Prior candidate `74da56c4` passed all required CI, including canonical typecheck and build, in [CI run 41929148958](https://github.com/paperclipai/paperclip/actions/runs/41929148958). Its fresh Greptile was 5/5. This is prior-head evidence. - The intermediate `053150d9` head also received Greptile 5/5. Its [CI run](https://github.com/paperclipai/paperclip/actions/runs/41940014165) and review stay retained as superseded evidence. They do not qualify the final catalog correction. - A clean full local typecheck/build is not claimed because the reused dependency cache has stale adapter-utils/Runner declarations. No source or lock workaround was made. Fresh canonical CI must pass before merge. - The original full-plan result remains 13/20. The separate two-case probe remains 1/2. A fresh complete saved-plan run against delivered source is required for acceptance. No case, assertion, model, budget, timeout or failed grade is changed. Focused commands: ```sh # From server/; database URLs point only to an owned disposable local instance. node ../node_modules/vitest/vitest.mjs run src/__tests__/chat-completion-proof.test.ts PAPERCLIP_TEST_DATABASE_URL="$OWNED_TEST_DATABASE_URL" node ../node_modules/vitest/vitest.mjs run src/services/native-runtime/confirmation-comment-api.integration.test.ts PAPERCLIP_TEST_POSTGRES_ADMIN_URL="$OWNED_TEST_DATABASE_URL" node ../node_modules/vitest/vitest.mjs run src/__tests__/confirmation-comment-resolution.test.ts ``` ## Risks - The ownership exception applies only to an agent acting as the current execution run against a different non-null source run. Human, unowned and current-source requeue behavior must stay intact. Native completion-review handling remains separate. - The completion proof fails closed on new native tools or event shapes. A new shape needs review. - A stored unchanged lease receipt is supporting local evidence. The final real journey must prove full revision, execution and delivery. - The new head still needs normal protected merge approval. The optional queued Paperclip review is not a passing review. No causal repair of unrelated CI intermittency is claimed. ## Model Used OpenAI `gpt-6.1-sol` in Codex, with `max` reasoning effort. The runtime reports a 475,000-token context window. Used reasoning, code editing, shell tools and provider-free controls. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
eab672e10a |
fix(workspaces): name the failing remote copy-back step without private output (#19152)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native run in a Daytona sandbox saves its workspace after the provider turn. The save is called copy-back. > - Copy-back fails on staging after successful provider turns. Issues become blocked with `retry_exhausted`. > - The recorded cause is only `remote_workspace_operation_failed` or `workspace_sync_out_failed`. Operators cannot see which step failed or why. > - The remote command wrapper drops every unrecognized sandbox failure to the bare code. The finalizer retry keeps only the disk-full cause. > - This pull request records a bounded, redacted cause: the operation, its exit code or signal, and the failing capture stage with structured error fields. > - The benefit is that the next copy-back failure shows its first cause, and sandbox output still cannot leak paths or credentials. ## Linked Issues or Issue Description **What happened?** Native runs on a Daytona sandbox finish their provider turn and accept their result. Then workspace copy-back fails. The first `workspace_finalize` operation records only `remote_workspace_operation_failed`. Each retry records only `workspace_sync_out_failed`. The run has no error text. After the retries stop, the issue is blocked. **Steps to reproduce** 1. Run a native task in a Daytona sandbox. 2. Make the sandbox capture step fail with an error that is not disk-full or a known archive validation error. For example, make Git exit with 128. 3. Read the run's workspace operations. **Expected behavior** The failed operation names the step that failed and a redacted cause. **Actual behavior** The failed operation shows only `remote_workspace_operation_failed` and then `workspace_sync_out_failed`. Refs #19050, #19059, #19061, #19063. These are related Daytona and finalizer repairs. This PR does not duplicate them. ## What Changed - `remote-workspace-snapshot-archive.ts`: add `remoteWorkspaceFailureDetail`. It is self-contained and runs in the sandbox. It reports only fixed keys (`code`, `exit`, `signal`, `syscall`, `killed`, `program=git`, `name`, `reason`). Each value is at most 64 `[A-Za-z0-9_]` characters and comes from structured error properties. It never reports message text, command lines, or paths. The archive child now writes `remote_workspace_archive_failed: <detail>` for an unclassified capture error. - `remote-workspace-transport.ts`: the helper-install and capture scripts write `remote_workspace_capture_failed: stage=helper|archive|publish <detail>`. The server accepts this text only when it matches a strict grammar. Unrecognized failures now throw `RemoteWorkspaceOperationError` with the message `remote_workspace_operation_failed: operation=<op> exit=<n>[ signal=<SIG>][; <validated cause>]`. Existing exact diagnostics (Git operation in progress, full disk, archive validation) do not change. - `native-workspace-finalizer.ts`: a retried remote save records the redacted cause in the operation stderr and in `metadata.workspaceSync.cause`. Other error text still stays off the record. - Tests: new red/green cases. The old privacy tests now assert the anchored redacted form, which still excludes any synthetic private text. ## Verification - Red, before the fix (only the error class existed). 8 tests failed on behavior: - The real Git failure produced `Received: "remote_workspace_operation_failed"`. - The signal-terminated command produced the same bare message. - The forged-cause cases produced the same bare message. - The finalizer retry case lost the cause. - Green, after the fix: `npx vitest run src/services/remote-workspace-transport.test.ts src/services/native-runtime/native-workspace-finalizer-failure.test.ts src/services/remote-workspace-snapshot-archive.test.ts` reported 126 passed. - The new capture test runs the real helper scripts with a controlled `git` that exits 128 and prints a synthetic private path. The error is exactly `remote_workspace_operation_failed: operation=capture exit=1; remote_workspace_capture_failed: stage=archive exit=128 program=git helper_exit=1`. The private path is absent. The workspace files are unchanged. The test passes with split output and with one-shot output. - Forged causes are dropped. These cases are tested: a path in a value, an extra line, an unknown key, and a prefixed text. - Wider sweep: I ran all 63 server test files that touch remote workspaces, copy-back, or finalization. 6381 tests passed. 2 tests failed in `chat-channels.integration.test.ts` and `git-workspace-expiration.test.ts`. These 2 tests also fail with this PR's three source files reverted to `master`. This PR does not cause them. - Typecheck: `tsc` reports no errors in the changed files. Other errors are local, from unbuilt workspace packages. - Not done: I did not run a live Daytona copy-back. This change adds diagnostics. It does not repair the underlying failure. The staging failures can be diagnosed after this PR is deployed. ## Risks - The error message for an unrecognized remote failure changes from the exact bare code to the same code plus validated fields. No production code compares the exact text. Only tests did, and I updated them. - Privacy: the sandbox detail contains only structured error properties. The server re-validates them with an anchored grammar with a length limit and a field limit. A path, a message text, or an unknown key fails the grammar, and the server drops the whole cause. - Retry and classification policy do not change. `classifyNativeWorkspaceFailure` does not read the new text. - This PR does not address two items: the final snapshot operation stays `running` after retries are exhausted, and no disk-retention cause was proven for the ACPX disk-full report. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, browser read-only GETs on staging, code execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
5b99bdae47 |
fix(native): verify requested publication at finalization (#19130)
## Thinking Path > - Paperclip manages work done by AI agents. > - Native execution reports work completion through a saved result. > - Some native providers can build that result from their final message. > - This path can bypass the finish tool's publication check. > - A task can then be marked done without the document the user requested. > - This change checks publication again at finalization and status commit. > - Missing output uses the existing bounded repair path with specific feedback. ## Linked Issues or Issue Description **What happened?** A native provider returned a final message about a local Markdown briefing. It had not published a task document or a verified attachment. Its synthetic result claimed success. The server accepted the claim and marked the task done. A real PostgreSQL regression reproduced this on public master. **Expected behavior** A requested task document must have valid saved publication evidence before the task becomes done. The check must apply even when the provider does not call the finish tool. A plain answer still needs no document. An explicit document-free fallback requires the actual human rejection for the current unchanged request. **Steps to reproduce** 1. Request a Markdown briefing document on a task. 2. Complete a native turn with a synthetic `provider:opencode:agent-message` result and no published output. 3. Admit the result through the normal control-plane port and finalize it. 4. The old code marks the task done. The repaired code keeps it open and queues one repair with publication instructions. **Paperclip version or commit** Reproduced from public master `7d1e9b3477f12780e987c32fb8191d476c210fda`. Related: #18661 added the finish-tool publication guard. This change applies that guard to finalization. #18180 covers a different fallback problem: no final text. This change preserves native provider fallback generation and its saved claims. ## What Changed - Run the existing deliverable guard before accepting a native done decision. Keep the original provider result unchanged. - Route missing publication through the existing single incomplete-result continuation. Include the publication feedback in the wake. A repeated incomplete result uses the existing named recovery action. - Inspect publication through a typed safe projection of the accepted ledger. Keep whole-protocol validation at result admission. An authoritative done decision must pass publication checks even when its accepted ledger is partial. - Recheck publication under the status-commit transaction. Hold only matching proof rows with non-waiting share locks. Changed or locked proof uses the existing bounded status replanning path. - Permit an explicit document-free decline fallback only for the saved human rejection selected by the current continuation. Require the same original request, current saved description, source revision and source contract. Model prose or an unrelated old decision is insufficient. - Add real database controls for publication, plain answers, authenticated decline, stale or wrong decisions, bounded recovery, and deletion between assessment and commit. ## Verification - The first regression failed before the repair: the synthetic result marked an unpublished task done. The retained real PostgreSQL run took 21.846 seconds, including setup and cleanup. - Two description-edit regressions also failed before their repair: a description change after admission or at the commit boundary still let the old decline waive the newly required document. Both now keep the task open. The red run took 25.738 seconds. - The commit-race regression also failed before its repair: removing publication after assessment still allowed done. That run took 17.448 seconds. - All thirteen focused real database controls pass after the repair. They exercise the normal control-plane port, finalizer, semantic publication tools, saved human rejection and status transaction. Owned database processes, data and output storage are removed after each run. - All 469 tests across the seven relevant and affected suites pass on head `f8870469f9879a53551f0faaff0fe5b1234ea303`, based on master `c7390bef20`. The six-suite run passed 426 tests in 97.319 seconds including owned setup and cleanup. The unchanged reassessment-worker suite passed 43 tests in 48.901 seconds. All required current-head checks pass in [CI run 41937653209](https://github.com/paperclipai/paperclip/actions/runs/41937653209), including aggregate verify, typechecking, build, general and serialized tests, native Runner tests and all eight browser shards. Fresh Greptile review is 5/5 on this exact head, with no open findings. - Local scoped server typechecking reported no errors in the four changed files and 11 errors in reused dependency declarations and exports. That setup failure remains retained. Fresh current-head CI passes the full Typecheck + Release Registry gate and Build. Local typecheck is not claimed as passing. - CI on the previous head found one shared status-projection regression across Studio, routine, status-authority and worker controls. Those phases use opaque accepted ledgers. The correction inspects only publication references, preserves the accepted marker and bindings, and forces the publication check for an authoritative done decision. A new real commit control verifies that an empty accepted ledger cannot close a requested-document task. All original affected assertions remain unchanged. The retained failed run is [41934314437](https://github.com/paperclipai/paperclip/actions/runs/41934314437). - No provider call, paid campaign, staging deployment or eval regrade is part of this PR. No model, timeout, assertion, workflow, dependency or lockfile change. Focused suites: ```sh ./node_modules/.bin/vitest run --config "$PWD/server/vitest.config.ts" --root server \ src/services/native-runtime/paperclip-control-plane-port.test.ts \ src/services/native-runtime/native-completion-feedback.test.ts \ src/services/native-runtime/native-deliverable-feedback.test.ts \ src/services/native-runtime/native-skill-test-status.test.ts \ src/services/native-runtime/native-routine-status.test.ts \ src/__tests__/native-status-arbiter-corpus.test.ts \ src/__tests__/native-status-reassessment-work.test.ts ``` ## Risks A native result that previously closed a task without its requested output now leaves it open. The existing repair path can run once and then records a named recovery action. Database contention does not grant completion. The existing first-task decision gate still runs first. The conditional decline check is deliberately conservative; unmatched wording or missing authoritative source data does not waive publication. This change closes a false completion path. It does not establish why a provider omitted publication, prove a live provider repair, or qualify a full evaluation plan. ## Model Used OpenAI Codex, based on GPT-6, assisted with diagnosis, code and controls. The exact serving model identifier and context-window size are not exposed in this session. Capabilities used: repository inspection, code editing, local code execution and GitHub tools. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal issue identity - [x] I have run relevant tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have documented the behavior and limits in this PR; no operator command or public API changed - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open findings - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandClaude Opus 5.5
|
b9fbbf8d30 |
fix(native): continue runs woken by withdrawn or cancelled interactions (#19163)
Withdrawing a card or cancelling a question wakes the assignee with that interaction id. The native interaction bridge required every wake interaction to be accepted, rejected or answered, so preparation failed with native_interaction_unresolved before the provider started. Project a cancelled interaction as a closed response with its stored result. Consumers grant authority only for accepted, answered or pending items, so the closed response carries context and no approval. Pending requests and governed tool reviews keep their existing checks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Devin FoleyandClaude Opus 5.5
|
7f9ef38273 |
fix(runner): report the upstream cause of OpenCode provider failures (#19166)
OpenCode reports some OpenRouter failures as an UnknownError whose message is the provider's JSON error body. The provider failure now reads its HTTP status, error type and message from that body, so the run shows the upstream message instead of raw JSON. The OpenCode nativeTurnFailure diagnostic now carries the provider failure's category, retryability and HTTP status (closed values only; no provider text or code), and recognizes the driver's opencode_empty_response code. Run status, error codes and retries do not change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Nicky Leach
|
737db73275 | fix(native): measure resumed run usage from the prior run's total (#19161) | ||
|
Devin FoleyandClaude Opus 5.5
|
7edbc1fc07 |
fix(server): keep a successor's issue lock when a finalizing native run releases its checkout (#19156)
`release_checkout` now clears `checkoutRunId` and the execution lock only when they belong to the committing run, so a successor that claimed the issue while its predecessor was finalizing keeps its claim and passes reviewed-chat attestation. `attestReviewedExternalChatRun` reports the first failed check as a closed label. Setup failures store it in `resultJson.reviewedChatBinding.denialReason`, and Sentry run-failure diagnostics export it as `execution.reviewedChatBindingDenialReason` (no IDs or text). The error message is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
482ec0aea9 |
fix(native): run saved copy-back retries on schedule behind a slow export (#19145)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs copy their sandbox workspace back after the provider finishes. A failed copy-back saves a retry 30 seconds later, with a bound of 3 attempts. > - One shared delivery queue runs these retries. A pass of that queue ends only when its slowest check ends. > - A legitimate sandbox export can take 10+ minutes. While it runs, all other due retries wait. > - On staging, a run active across a deploy showed "Reconnecting" with an overdue retry for about 24 minutes. Its retries ran at +24 min and +3.5 min instead of +30 s. It settled only after about 32 minutes. > - This pull request lets idle check slots admit entries that come due during a pass. It does not overlap a running check's resource and does not start work after abort or drain. > - The benefit is that the existing 3-attempt bound settles a failing copy-back in about 1 minute, not after the slowest unrelated export. ## Linked Issues or Issue Description **What happened?** A native run whose result is durable but whose workspace copy-back failed stays in `running` with the label "Reconnecting" for a long time. Its saved retry time is in the past. The retry does not run until an unrelated slow copy-back in the same queue pass finishes. **Steps to reproduce** 1. Start two native runs on different tasks with accepted results. 2. Let the copy-back of run A take many minutes. 3. Let the copy-back of run B fail. It saves a retry 30 seconds later. 4. Watch run B. Its retry does not run until run A's copy-back ends. **Expected behavior** Run B retries on its saved schedule. After 3 failed attempts it settles with `native_workspace_sync_out_retry_exhausted` and a recovery action, while run A continues. **Paperclip version or commit** master at |
||
|
|
c2cc69d1e4 |
fix(recovery): resume and settle preserved native results (#19034)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native execution can save an accepted result before workspace export fails. > - Recover must preserve the files and settle that same result without another agent turn. > - The export failure can own the task's Blocked transition and its recovery notice. > - Recovery must prove preservation, stop, source identity and current authority before settlement. > - Recover can also transfer repair ownership to the system, which must keep explicit board export repair available. > - This change restores those paths while preserving later task state, live owners and unknown action outcomes. ## Linked Issues or Issue Description Related #17916 protects unfinished exports. Related #19004 preserves deliberate Retry authority. This change repairs accepted-result finalization and its recovery notice. **What happened?** A native run accepted its result, then exhausted workspace export. Normal Recover could save the files but leave the result and old recovery notice unsettled. A saved Retry could also use a different generation from preservation. Separately, Recover changed the action owner from board to system. Explicit export Retry then rejected that same source as stale, even when its accepted result and stopped allocation still matched. **Expected behavior** Preserve files, finish the original accepted result and close only its obsolete notice after exact proof. Keep explicit board export repair available after system adoption. Preserve current authority, Stop, later task changes, newer execution and every physical ownership fence. **Steps to reproduce** 1. Accept a native result and exhaust its workspace export. 2. Request ordinary Recover, optionally with an existing saved Retry. 3. Let file preservation complete. Inspect the task and source recovery notice. 4. To reproduce the explicit-repair defect, call export Retry after Recover adopts the action. Before this repair it rejects the valid system-owned source before any provider call. ## What Changed - Record the exact accepted result, contract, task status version and prior decision when export exhaustion creates Blocked. - Bind passive finalization to complete, source-bound preservation and the current chat/review authority. Recheck the proof under status commit locks. - Keep the saved Retry identity separate from its preservation generation. Retain coalescing and reject malformed or mismatched history. - Preserve an existing published GitHub assessment for the no-publication status-unproven wait. Keep all semantic, result, authority and publication checks. - Close an old export notice only after its own accepted result, export or preservation receipts, effects, projection and old Runner ownership settle. Keep newer work and files unchanged. - Permit explicit board export Retry after Recover transfers the exact source to the system. Require valid system and workspace recovery records for that same run, plus every existing result, stop, allocation, task and shared physical-lock check. Reject missing proof, another source and agent ownership before provider access. - Merge current master while retaining the stronger Stop, source and verified preservation checks. - Update the execution contract. No schema, endpoint, Runner protocol, model, trust or provider artifact change. ## Verification - Current head: `9be5a73489801095c9f39c9bfcb285a22b055912`. - The explicit-repair regression fails before the new guard change: ordinary Recover adopts the real database action, then export Retry rejects it as stale. It passes after the repair. - On the merged source, three owning PostgreSQL/file suites pass **220 tests**: `preserved-native-response-wait.integration.test.ts`, `native-workspace-export-retry.test.ts` and `idle-completed-history.test.ts`. The export suite includes four unrelated-owner/source rejection controls and both board-owned and system-adopted active-exporter controls. Results and run counts prove that repair creates no provider turn. - Server layered typecheck, module boundaries, `git diff --check`, and the added-line secret/private-incident scan pass. - Full workspace typecheck (`pnpm -r typecheck`) and build (`pnpm build`) pass. The complete local `pnpm test:run` is still running; its completed serialized chat partitions pass. Current-head CI is green (53 successful checks, 3 expected skips). Greptile reviewed `9be5a73489801095c9f39c9bfcb285a22b055912` at 2026-11-05 00:58 UTC with confidence **5/5**; all review threads are resolved. - Original qualification remains supporting evidence: the accepted-result fixture exercises HTTP Recover, real files, the finalizer, saved Retry coalescing, eight commit races, current principal/resource/head changes, Stop, later task changes, notice closure and repeated delivery. No accepted response, provider turn or publication is replayed. - Earlier failures remain recorded. Original red controls exposed wrong-action preservation, unrelated finalization barriers, generation mismatch and a lost GitHub assessment; the original repair added controls for them. The prior CI run [41908572248](https://github.com/paperclipai/paperclip/actions/runs/41908572248) failed an unchanged completed-history runtime fixture because its loopback service refused connections through the original ten-second readiness window. That owning suite passes unchanged in the 220-case run above. The earlier readiness failure's cause remains unproven; no wait or assertion was weakened. - Local and CI tests do not establish hosted task recovery. The serving revision and original-task outcomes require separate verification after rollout. ## Risks - Historical failures without the exact export-owned Blocked receipt keep their current task status. A saved result does not grant execution or publication authority. - Explicit export repair still resumes only the same retained allocation under the shared physical lock. It cannot take over an active exporter, create another agent turn, choose a replacement sandbox or discard saved files. - Retiring a source notice does not release physical resources or modify a newer task owner, later Done state or saved snapshot history. - The proof adds bounded reads only for concrete recovery obligations. It adds no recurring worker or polling timer. ## Model Used OpenAI GPT-6 through Codex for the original repair. OpenAI GPT-6.1 Sol (`gpt-6.1-sol`), xhigh reasoning, for the merge, integration verification and system-adopted export repair. Both used repository tools, code execution and PostgreSQL tests. The runtime does not expose a context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used - [x] I have checked ROADMAP.md and searched for duplicate or related work - [x] I have described the issue and linked related public PRs - [x] I have included no internal task identifiers or private incident links - [x] I have added behavioral and safety regression tests - [x] I have documented verification limits and risks - [x] I have updated the execution contract - [x] All Paperclip CI gates are green on the current head - [x] Greptile is 5/5 on the current head with no unresolved comments - [x] I will address review comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Devin Foley <devin@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
11f2a5293d |
fix(native): name the proof that holds a recovered per-turn owner (#19061)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native Runner per-turn sessions in a remote sandbox keep one conversation across task turns. > - A follow-up must prove that the previous Runner owner is finished before it starts a new Runner. #18846 added a retirement path for an owner that finished without a saved Runner checkpoint. > - On a test deployment, follow-ups still fail with `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate`. The retirement path has about twenty separate fences, and each refusal gives the same error. > - The read-only run data cannot tell which fence held these runs. A code fix without that evidence would be a guess. > - This pull request names the first failed retirement proof in the error. Admission behavior does not change. > - The benefit is that the next failure on a deployment identifies the exact fence, so the follow-up fix can target it. ## Linked Issues or Issue Description Refs #18846 (retirement of recovered per-turn owners). Refs #18992 (settlement from live state). **What happened?** On a test deployment, 16 runs in one day failed before provider work with `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate`. All were per-turn `acpx_runtime` runs in a Daytona sandbox, and each got a new sandbox. In the runs that I inspected, the previous run committed its result. In one of them, `harness_state.backup.persist` and `session.checkpoint.persist` ran for about 346 seconds and failed, so the Runner left no suspension receipt or failover backup. That is the case that #18846 addresses, but the follow-ups still failed. The run log does not show the retirement message, and the error is the same for each fence. Thus the cause cannot be found from the run data. **Expected behavior** When admission holds a follow-up, the error names the proof that failed. **Steps to reproduce** 1. Seed a per-turn sandbox owner that committed its result without a Runner suspension receipt. 2. Make one retirement proof fail. For example, leave the owner phase `observed`, or leave a `runner.suspend` command pending. 3. Admit a follow-up in a new sandbox. 4. Before this change, every case fails with the same error text. **Paperclip version or commit** Observed on a deployment before `e107a75755`. The same error text is on `master`. ## What Changed - `retireFinishedPerTurnRunnerdOwner` returns the refusal reason instead of `false`. `readSettledRunnerdProjection` returns the journal refusal reason instead of `null`. - When admission tries the retirement and it fails, the error is `prior_owner_terminal_state_indeterminate; retirement_refused=<reason>`. A warm session gets no suffix, because retirement does not apply to it. - Reasons use only fixed lists of known names: owner phase, recovery state, controller fence reason, unsettled command type and status, sandbox outcome, and quarantine copy result. Any other stored value appears as `other`. Reasons do not include payloads, file names, or paths. - The same proofs apply in the same order. Admission accepts and refuses the same states as before. - `doc/execution-semantics.md` describes the suffix. ## Verification - **Failure before the change:** I ran the updated `native-recovered-owner-followup.test.ts` against the unchanged executor. 18 tests failed. Example: expected `...indeterminate; retirement_refused=owner_observed`, received `...indeterminate`. - **After the change:** `cd server && pnpm exec vitest run src/services/native-runtime/native-recovered-owner-followup.test.ts`: 22 passed. Each held case now asserts its reason: `owner_retryable_failure`, `controller_remote_controller_lease_live`, `owner_recovery_resuming_session`, `journal_commands_unsettled(runner.suspend.pending)`, `journal_tool_input_unanswered`, `quarantine_journal_identity_mismatch`, `runner_lifecycle_not_suspended`, and `quarantine_runner_lifecycle_not_suspended`. - The success cases still retire the owner and admit the follow-up. - `pnpm exec vitest run src/services/native-runtime/native-session-executor.test.ts src/services/native-runtime/native-runner-restart-recovery.integration.test.ts src/services/native-runtime/remote-warm-session-recovery.test.ts`: 1227 passed, 20 skipped. - `pnpm exec vitest run src/services/native-runtime/native-controller-recovery.integration.test.ts src/__tests__/task-workspace-binding.test.ts`: passed. - `tsc --noEmit` reports no errors in the changed lines. This worktree did not build every workspace package, so the full-package typecheck shows unrelated missing-module errors. CI runs the full typecheck. - **Not verified live:** no deployment has run this change. ## Risks - Low risk. Admission logic is unchanged. Only the error text gets a suffix. - Consumers that match the error by prefix still match. `failed-run-retry.ts` uses `startsWith("runner_state_identity_mismatch: ")`. No code compares the full string outside tests. - This change does not fix the held runs. It finds the fence. A second change is needed after a deployment reports the reason. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use, shell execution, and read-only browser access to a test deployment. The harness does not expose the context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
3a72efebe5 |
fix(native): confirm no runner process when remote staging fails before launch (#19059)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native sandbox runs supervise a remote `paperclip-runnerd` process. When that process exits during a turn, the transport restarts it inside a reconnect grace window. > - The server launcher stages the runtime assets first. Then it sends one launch RPC that starts runnerd. > - When staging failed, the launcher reported the exit as unconfirmed. This happened although no launch RPC was sent and no process could exist. > - Reconnect recovery treats an unconfirmed exit as a possible second owner. So it stopped at once with `runner_process_termination_unconfirmed` and did not retry. > - This pull request confirms the exit for failures before the launch RPC. Failures after dispatch stay unconfirmed. > - The benefit is that a transient staging failure, such as a full sandbox disk, uses the normal bounded retry. A real failure ends with `transport_reconnect_grace_exceeded` and keeps the staging error as the detail. ## Linked Issues or Issue Description **What happened?** On a hosted staging deployment, 7 native ACPX runs failed in 24 hours with `Provider notification transport failed closed.` (`native_provider_terminal_failed`). In every run, the runner process exited during the turn. The replacement launch then failed while it uploaded the runtime context into the sandbox: `runner_process_termination_unconfirmed: replacement runner health could not be verified: ... .paperclip-upload-<uuid>.tar.gz: create: ... no space left on device`. The sandbox disk was full. No replacement process was started. But the launcher said that it could not confirm the exit. So recovery did not retry inside its grace window. **Expected behavior** A staging failure before the launch RPC proves that no replacement process exists. Recovery must retry within the configured reconnect grace. If the failure continues, the run must end with the reconnect-grace failure and the staging error. **Steps to reproduce** 1. Create a remote runner launcher whose `ensureArtifact` step throws, for example with an ENOSPC upload error. 2. Wait for the handle completion to reject. 3. Before this change, `isExitConfirmed()` returns `false`, and no launch RPC was sent. The transport then fails closed with `runner_process_termination_unconfirmed`. **Paperclip version or commit** Observed on staging revision `89703c3`. Reproduced on `master` at `be20d4ad97`. Related: #19035 and #19014 change other runner paths (replayed terminal turns and redacted external tool results). They do not change remote launcher exit evidence. #18999 adds a free-space reserve for host-volume seed copies. It does not change sandbox disk use or this classification. ## What Changed - `createRemoteRunnerProcessLauncher` records when it dispatches the launch RPC. If the completion fails before that point, the handle reports `isExitConfirmed() === true`. The original error is kept unchanged. - A failed or timed-out launch RPC still reports an unconfirmed exit, because the RPC may have started runnerd. - Added a launcher regression for a staging failure before dispatch. - Added safety cases for a failed and a timed-out launch RPC. - Added a transport regression with the real `paperclip-runnerd`. A replacement whose staging fails is retried, and the next launch restores the durable session without a notification failure. ## Verification - Red before the fix: `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts -t "remote runner process supervision"` failed `confirms no replacement process exists when staging fails before the launch RPC` with `expected false to be true`. The two launch-RPC safety cases passed. - Green after the fix: `npx vitest run src/services/native-runtime/native-session-executor.test.ts` passes 1131 of 1131 tests. - `npx vitest run src/services/native-runtime/setup-readiness.test.ts` passes 98 of 98 tests. - `cd packages/paperclip-runner && npx vitest run src/live/runnerd-codex-transport.test.ts` passes 246 of 246 tests with the real debug `paperclip-runnerd`. This includes the existing case where a replacement with a dispatched but unverifiable process still fails closed. - `npx tsc --noEmit -p server` and `pnpm --filter @paperclipai/paperclip-runner typecheck` pass. - Not run: a live hosted sandbox with a full disk. The staging data showed the exact error, but this change was not deployed there. ## Risks - Low risk. The change affects only failures that occur before the launch RPC. In that window, the launcher has not run any sandbox command that can start runnerd. - Retries now run for the reconnect grace window, which is bounded and uses the existing backoff. Each failed staging attempt cleans its own upload scratch. - This change does not free sandbox disk space. If the disk stays full, the run still fails, now with `transport_reconnect_grace_exceeded` and the ENOSPC detail. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use, shell execution, and code editing. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
5723fd82ca |
fix(native): release a fresh claim when task preparation fails (#19065)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native run claims its finalization coordinator before task preparation. Then it starts the provider turn. > - Task preparation can fail after the claim and before any provider turn. > - In that case the executor kept its claim. Failure settlement read the claim as a live owner and stopped. Nothing failed the run. > - The 60-second execution-control watchdog then failed the run as `execution_finalization_deadline_exceeded`. The original error was lost. > - This pull request releases only that exact fresh, unstarted claim after renewal stops. > - The benefit is that ordinary settlement fails the run at once with its real error and keeps the workspace files. The watchdog stays a backstop. ## Linked Issues or Issue Description Refs #19028. That PR keeps the earlier failure code when the watchdog fails a run. In this case no earlier code is saved, so it cannot help. This PR removes the wait for the watchdog. I searched open PRs for duplicates. #18820, #19034 and #19038 touch near code, but none release a preparation claim. **What happened?** A native task with a prepared sandbox workspace failed during preparation, after the controller claim committed. The coordinator stayed `observed` and kept this controller's lease owner. The heartbeat failure path calls workspace preservation. Preservation refuses any lease owner without a matching restart claim, so it raised `NativeWorkspaceFinalizationBusyError`. The heartbeat then treated ownership as held and returned without a terminal write. About 60 seconds later the watchdog failed the run with a generic "verify the provider stopped" action. The real error (for example `native_harness_storage_owner_missing`) stayed only in the workspace operation record. **Expected behavior** When no provider turn started under a fresh claim, settle the failure like a failure before the claim: retain the source, fail the run with its own error, and do not wait for the watchdog. **Steps to reproduce** 1. Start a native run with a prepared remote workspace. 2. Make task preparation fail after `native.coordinator.claim` commits (for example, a preparation callback throws). 3. See that the run stays `running` until the control deadline fails it as `execution_finalization_deadline_exceeded`. ## What Changed - `native-execution-lease.ts`: add `releaseUnstartedNativeExecutionClaim`. It clears the lease owner and expiry only for the exact run, company, issue, owner and attempt, while the phase is `observed` and no result exists. - `native-session-executor.ts`: when task preparation fails after a fresh claim, mark that claim. The startup owner releases it after renewal stops. A restart-recovery claim, a startup-cleanup context, Stop, restart detach, or a different owner keeps the old behavior. - Tests: an executor regression for the fresh and restart cases, and a real-database test for release and workspace retention against the exact owner, a foreign owner, a different attempt and an accepted result. - `doc/execution-semantics.md`: state that the deadline is a backstop and describe the release. ## Verification - Red before the fix: `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts -t "releases only a fresh claim"` fails with "releaseUnstartedNativeExecutionClaim ... Number of calls: 0". - Green after the fix, with the same command. - `npx vitest run src/services/native-runtime/native-session-executor.test.ts src/services/native-runtime/native-failed-workspace-recovery.test.ts src/services/native-runtime/native-execution-lease.test.ts src/services/execution-control-reconciliation.test.ts` passes (embedded Postgres for the retention test). - `pnpm --filter @paperclipai/server typecheck` passes. - Not done: a live run against a real sandbox at this revision. The staging evidence is read-only. ## Risks - The release is fenced by the exact owner, attempt, `observed` phase and no result. Renewal stops before the release, so a renewal cannot see a missing owner and detach or cancel the run. - The fix covers failures inside task preparation only. Failures later in startup, or after a restart-recovery claim expires, still reach the watchdog. Those keep fail-closed behavior and need separate evidence. - If the release write fails, the run keeps the old watchdog path. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
cda9460ae9 |
fix(native): restart remote bootstrap after a shutdown during preparation (#19056)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native runs in a remote sandbox go through workspace preparation
before the Runner process starts.
> - When the server restarts, restart recovery decides if a remote run
can safely restart its bootstrap.
> - A remote run that never started a Runner process, never had a lease
owner, and has no provider evidence can restart its bootstrap. Recovery
calls this an "unclaimed bootstrap".
> - A graceful shutdown (SIGTERM) suspends every running native
coordinator. It sets `leaseExpiresAt = now`, also when the coordinator
has no lease owner.
> - The unclaimed-bootstrap check requires `leaseExpiresAt` to be null.
So a deploy during workspace preparation turns a safe bootstrap restart
into `native_restart_recovery_blocked` /
`ambiguous_remote_recovery_evidence`. The task is blocked and a board
recovery action is created.
> - This pull request lets the check accept an elapsed expiry when the
coordinator has no lease owner.
> - The benefit is that a deploy during remote workspace preparation no
longer blocks the task. Every other contradictory-evidence guard stays
in force.
## Linked Issues or Issue Description
**What happened?**
On a staging instance, three remote-sandbox runs failed with
`native_restart_recovery_blocked` and
`ambiguous_remote_recovery_evidence` after one deploy. Each run was in
native workspace preparation (`snapshot.git`, `pack`, `seed.verify`,
`stage.workspace`). No run had a process start, a process identity, a
provider event, or a checkpoint. The graceful shutdown event was the
only coordinator change. The next server then blocked each run in
restart recovery.
**Expected behavior**
A remote run that was suspended during preparation, before any Runner
launch or lease claim, restarts its bootstrap on the same run. Recovery
does not block the task.
**Steps to reproduce**
1. Prepare a remote native run (prepared native workspace sync, fenced
preparation, no process, no checkpoint).
2. Call `drainRunningRunsForShutdown("SIGTERM")` for the run.
3. Call `claimNativeRestartRecoveries` as a successor controller with
`restartKind: "hard"`.
4. Observe `{ kind: "blocked", reason:
"ambiguous_remote_recovery_evidence" }`. Without step 2, the same run
gets `bootstrap_incomplete`.
**Paperclip version or commit**
Reproduced on `master` at `be20d4ad97`.
**Deployment mode**
Remote sandbox environments with PostgreSQL.
Related: #18877 (remote checkpoint retry reclaim) and #18854
(pre-provider startup retry). Neither changes this predicate.
## What Changed
- `claimNativeRestartRecoveries`: the unclaimed remote bootstrap check
accepts `leaseExpiresAt` when the coordinator has no lease owner and the
expiry has elapsed. A null owner with an elapsed expiry grants no
authority.
- New regression: a remote run suspended by the real graceful-shutdown
drain during preparation is claimed as `bootstrap_incomplete`, with no
recovery action.
- New safety case: an ownerless coordinator with an unexpired lease
expiry still blocks with `ambiguous_remote_recovery_evidence`.
## Verification
- Red before the fix: `npx vitest run
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts
-t "graceful shutdown suspended during preparation"` returned `blocked`
/ `ambiguous_remote_recovery_evidence`.
- Green after the fix: the same test returns `bootstrap_incomplete` with
`providerAttempt: 0` and `controllerGeneration: 1`.
- The existing contradictory-evidence cases (`claimed`, `generation`,
`process`, `launch`, `provider-event`, `checkpoint`,
`invalid-workspace`, `unfenced`) and the new `unexpired-lease` case
still block.
- Full `native-runner-restart-recovery.integration.test.ts` suite and
server `tsc --noEmit` pass locally.
- Not run: a live deploy on a remote sandbox. The fix is not deployed.
The original blocked tasks are not recovered by this change.
## Risks
- Low risk. The change relaxes one field of one predicate. All other
unclaimed-bootstrap conditions stay: prepared workspace, fenced
preparation, attempt 0, generation 0, no controller identity, no lease
owner, no process identity, no session, no checkpoint, no provider
evidence, and no launch or tool event.
- The executor still requires the durable PRP root to prove an
incomplete bootstrap before it starts a fresh bootstrap.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with
tool use (shell, file edits, browser reads of staging). Standard context
window.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandPaperclip
|
7d46532c6d |
fix(runtime): bound parallel offline harness backup transfers (#19108)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - An ephemeral web server must retain provider history outside its
local disk.
> - Offline enrollment moves verified local harness backups to object
storage.
> - A cold rehearsal reached 699 current backup rows while processing
1,953 sources before its 25-minute deadline.
> - The serial transfer loop prevents this existing dataset from
enrolling.
> - This pull request overlaps at most four transfers within the
existing total scratch limit.
> - The benefit is faster preservation with the same ownership and
publication checks.
## Linked Issues or Issue Description
Refs #19038, #18943, #18932.
**What happened?**
Offline enrollment processed 1,953 local provider backups one at a time.
The normal Cloud image reached 699 current backup rows (including nine
pre-existing rows) before maintenance expired. The serving instance
remained unchanged during the isolated test.
**Expected behavior**
Enrollment must preserve every source within bounded time and disk use.
It must stop all in-flight transfers before it reports failure.
**Steps to reproduce**
1. Prepare a settled instance with multiple verified native harness
backups.
2. Run offline enrollment against shared object storage.
3. Observe that the serial owner waits for each archive, upload,
readback and publication before starting the next source.
## What Changed
- Process groups of up to four verified offline harness backups.
- Reserve space for the archive and encrypted copy before each group
starts. Keep the total scratch limit equal to one maximum-size serial
transfer.
- Derive a smaller archive limit from a closed manifest directory set.
Preserve unlisted files with an exclusive maximum-size reservation.
- Enforce the reservation while streaming. Keep the existing maximum for
ordinary runtime backups.
- Cancel and await peer transfers after failure. Keep completed
publications valid for retry.
- Preserve the original archive-stream failure when the tar writer also
fails after destruction.
- Document the migration bound and add integration regressions for
overlap, restoration, cancellation, retry, extra files and archive
limits.
## Verification
- Serial baseline: the new overlap regression fails with one active
transfer when four are required.
- Candidate: 59 targeted tests pass with real PostgreSQL, S3, filesystem
archives and restored provider bytes. The bundled-entry test is skipped
locally because this source worktree has no runtime bundle. The full
normal image rehearsal remains required.
- Follow-up: 48 archive/storage checks pass after correcting the
PAX-header allowance. The real encoder produces a 972,801,024-byte
full-entry projection, above the old 805,306,368-byte allowance and
below the new 1,024,001,024-byte bound. The existing global object limit
remains in force. The old formula fails the new regression. Exact-size
restoration and one-byte-over-limit rejection pass.
- The previous revision passed full CI after one setup-route timeout
retry. That failure remains recorded; the corrected current revision
|
||
|
|
ea6bd15e2a |
fix(native): record why finalization failed a run (#19063)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - A native run finishes in two steps: the provider turn, then
finalization. Finalization saves the workspace and commits the result.
> - When finalization exhausts its retries, `recordRetryableFailure`
sets the run to `failed` and blocks the task.
> - That code writes the cause only to `resultJson.failureCode`. It does
not write the run's `errorCode` or `error`.
> - So the run list, inbox, and agent status show a failed run with no
reason. Operators cannot tell why the run failed.
> - This pull request writes the finalizer's terminal failure code and a
readable message to the run row when the finalizer fails the run.
> - The benefit is that every failed run now names its cause.
## Linked Issues or Issue Description
**What happened?**
A hosted instance had 11 assigned native runs with status `failed`,
`errorCode` null, and `error` null in one day. In each run, the provider
turn succeeded. The remote workspace copy-back then failed three times.
The finalizer set the run to `failed` and blocked the task. The run kept
no error code. `resultJson.failureCode` was
`native_workspace_sync_out_retry_exhausted`, but no screen shows that
field as the failure reason.
**Expected behavior**
A failed run has an `errorCode` and an `error`. The failure surfaces
show why the run failed.
**Steps to reproduce**
1. Create a native run with a finalization coordinator.
2. Call `recordNativeFinalizationFailure` three times with
`native_workspace_sync_out_failed` and `failureScope: "workspace"`.
3. Read the run. Its status is `failed`, and its `errorCode` and `error`
are null.
**Paperclip version or commit**
`master` at
|
||
|
DottaandPaperclip
|
818d2bb661 |
fix(native): preserve closed post-result failure diagnostics (#19126)
## Thinking Path > - Paperclip manages AI agents and their work. > - Native execution persists a result before it completes cleanup and workspace export. > - A later execution error enters the failure-write path. > - That write must reject an already selected result. The existing code reports lease loss for every rejected write. > - This can hide the category of the first error even when the same controller still owns an unexpired lease. > - This change adds closed diagnostic labels after exact ownership and accepted-result checks. It keeps the existing control error and recovery policy. ## Linked Issues or Issue Description **What happened?** A local real-PostgreSQL regression accepts a result through the production control plane, then injects a harmless filesystem failure. The production executor reports `native_session_lease_lost`. It loses the original failure category because the failure transition requires a null result ID. **Expected behavior** Keep the result guard and the existing lifecycle classification. Record a closed diagnostic category when the selected result is accepted and the controller still has exact, unexpired authority. Keep ordinary lease-loss fencing when that proof is absent or its diagnostic lock is busy. **Steps to reproduce** Run the new post-result cases in `server/src/services/heartbeat/runtime-execution.test.ts`. They use the real executor, control-plane result commit, heartbeat recovery path and PostgreSQL. The provider boundary is controlled. No live provider is used. **Paperclip version or commit** The regression fails on base `7d1e9b3477`. Related open PR #16917 changes legacy lease recovery. Open #19063 adds terminal finalizer reasons, and #15310 preserves database finalization errors. They do not change this executor diagnostic branch. This change only adds native diagnostic evidence and does not reclaim a lease. ## What Changed - Add a typed diagnostic with a closed stage, category and accepted-result flag. Do not retain the source message, path, arbitrary code or lease token. - After a rejected failure write, check the selected accepted result and the exact company, task, run, owner, attempt, generation, process identity and expiry. Use a coordinator-only `NOWAIT` lock. Fall back to the original control error if the diagnostic read fails. - Report the closed event after rollback. Keep `native_session_lease_lost` as the error message and leave every failure-write predicate unchanged. - Add real-PostgreSQL safety cases for stale and expired authority, owner replacement, Stop state, result selection, pending accounting, one provider invocation, closed labels and diagnostic sink failure. - Document the closed event, unchanged authority, and possible late diagnostic sink completion. - Add a competing-row-lock barrier. It verifies actual PostgreSQL lock refusal and control settlement before the competing lock is released. ## Verification - Permanent regression on unchanged base: RED because the closed diagnostic is absent. - NOWAIT mutation: RED when only NOWAIT is removed. PostgreSQL reports lock timeout instead of immediate lock refusal. The test uses barriers and the actual database error, with no elapsed-time assertion. - Final related suites: 56 tests passed across runtime execution and provider failure integration tests. - Full server source and test typecheck: passed. - Full `pnpm -r typecheck`: passed in an independent locked-dependency checkout. - Full `pnpm build`: passed. - Full `pnpm test:run`: running in the independent immutable checkout. All six chat shards passed; the remaining suites are still pending. - The first current-head CI attempt failed in two cases: workspace runtime initial fixture HTTP readiness and instructions-bundle error-message rendering. Both exact cases passed in isolation on unchanged PR head and base. The original causes remain unknown after bounded test-only observation. One controlled same-head rerun of each failed job passed. Current required CI and all other completed checks passed. Original failures remain recorded; no infrastructure or flaky-test claim is made. - Module boundary check and `git diff --check`: passed. - Exact-head Greptile: 5/5 with no actionable findings or open review threads. Automatic native review remains queued. That advisory check is not required by current master rules; no current-head native assessment is claimed. - Earlier fixture setup failures remain recorded. These included missing fixture fields and observing the Drizzle wrapper instead of its PostgreSQL cause. They are not the regression RED. An earlier broad typecheck was interrupted for the required NOWAIT revision; the final full typecheck passed. - No hosted export recovery or original hosted failure cause is claimed. ## Risks The extra read is diagnostic only. It uses NOWAIT and cannot wait for a competing coordinator lock. Its failure preserves the existing control classification. The event sink uses the existing 60-second execution-control bound. A slow sink can finish later and emit a late event. Result selection, Stop, accounting, retry eligibility and provider work do not depend on event success. No schema or lifecycle contract changes are required. ## Model Used OpenAI GPT-6.1, model ID `gpt-6.1-sol`, with reasoning, tool use and code execution. The host did not expose the context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
35d086213e |
fix(native): explain failed accepted results on the run (#19091)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native run ends when the server projects the accepted result onto the run row. > - Operators read `errorCode` and `error` on a failed run to learn why it failed. > - Three native writers can set a run to `failed` without these fields. #19063 repairs the finalizer retry-exhaustion writer. > - The two result-projection writers still set `failed` with no cause when the provider failure is not classified. > - This pull request makes both projection writers record a cause when they project `failed`. > - The benefit is that, with #19063, no native writer can leave a failed run unexplained. ## Linked Issues or Issue Description Refs #19063. **What happened?** A staging soak found 11 failed native runs with `errorCode` null and `error` null. In each run the provider turn succeeded. Workspace copy-back then failed three times, and the finalizer retry-exhaustion path set the run to `failed`. #19063 fixes that writer. An audit of every `heartbeat_runs` status writer found two more writers with the same defect. `finalizeNativeRun` and `projectCommittedRun` project an accepted result with `runTerminalState: "failed"`. They write the error fields only when a classified provider failure exists. **Expected behavior** A run with status `failed` always has an `errorCode` and an `error`. A cause that is already recorded stays. **Steps to reproduce** 1. Accept a native result with `runTerminalState: "failed"` and no `turn.failed` event. 2. Call `finalizeNativeRun` with `projectRunStatus: true`, or project the committed run through `projectCommittedRun`. 3. Read the run. On master its status is `failed`, and its `errorCode` and `error` are null. ## What Changed - Add `explainFailedRun` in `server/src/services/native-runtime/native-run-failure-explanation.ts`. It builds SQL SET expressions. They keep a non-empty existing `errorCode` or `error`. Otherwise they write `native_run_failed` and a readable message. By default the expressions leave a run that is already `cancelled` unchanged, as the status writers do. - `finalizeNativeRun` uses the helper when it projects `failed`. The existing provider-failure and recovered-failure values remain the preferred causes. - `projectCommittedRun` uses the helper for a `failed` accepted result. This writer sets status unconditionally, so it does not use the cancelled branch. - Writer audit: all legacy and heartbeat callers of `setRunStatus*` that pass `failed` already set both fields. The other direct native writers also set both fields. ## Verification - Red, with the source change stashed: 3 new cases failed with `errorCode: null`. These were the finalizer projection, recovery projection with no cause, and recovery projection with an empty error. The control case with a recorded cause passed. - Green: `npx vitest run src/__tests__/native-run-projection-work.test.ts src/services/native-runtime/native-run-finalizer-telemetry.test.ts src/__tests__/native-status-arbiter-corpus.test.ts` (225 passed). - Wider run: all `server/src/__tests__/native-*.test.ts` and `server/src/services/native-runtime/*.test.ts` (7416 passed). Two failures come from the local environment. `plugin-runner-probe` needs the Rust runner source. `native-ingress-bootstrap-retry` needs the controller-host OpenCode proxy. - This change was not exercised on staging. It changes only fields that are written at terminal projection. ## Risks - Low risk. The change affects only rows that are projected to `failed`. A recorded cause is never replaced. Succeeded and cancelled projections are unchanged. - The staging failures need #19063 too. This PR alone does not change their writer. - The cause behind the staging copy-back failures is still unknown. The sandbox command reports the redacted `remote_workspace_operation_failed`, which also appears in a separate signature. This PR does not change that diagnostic. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, file edits, browser reads of staging run records). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Nicky LeachandClaude Opus 5.5
|
aa8372fd5d |
fix(accounting): post late usage to its owning run (#18991)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native Runner sessions keep one provider conversation across task turns, so Paperclip accounts usage per run. > - Codex can report more usage for a turn after its run's charge was posted. > - The Runner then marked the session `reconciliation_required`. If another run was attached, it also failed that run. Core refused the delivery with `native_usage_after_finalization_requires_reconciliation`. > - That usage has already happened. Blocking the session cannot undo it, and the debt waited for an operator while the run totals stayed wrong. > - Codex totals are cumulative, and the Runner already knows the owning run in most cases. So the correct amount can be posted automatically. > - This pull request reports the exact increment with its owner, and posts it to the owning run as a separate cost event. > - The benefit is that run accounting becomes correct without stopping work, and an old run's usage can no longer fail a newer run. ## Linked Issues or Issue Description No issue exists. I searched open and closed pull requests and issues for "late usage", `native_usage_after_finalization_requires_reconciliation` and `codex_usage_accounting_integrity`. I found no duplicate. **What happened?** Code review found this while diagnosing session recovery failures. When Codex reported usage above the known total for a turn that was not active, the Runner blocked the session. If that turn belonged to a previous run, the Runner also failed the attached run with `codex_usage_accounting_integrity`. Core kept the receipt as debt, and could correct it automatically only if the token counts were unchanged. **Expected behavior** Late usage does not stop any run or session. The charge goes to the run that used the tokens, and it is auditable. **Steps to reproduce** 1. Finish a Codex turn on a native Runner session and attach the next run. 2. Have Codex send `thread/tokenUsage/updated` with a higher total for the finished turn. 3. On the unfixed code, the attached run fails and later attaches are refused. `late_codex_usage_requires_durable_reconciliation_before_or_after_handoff` encoded this behavior. **Paperclip version or commit** `master` at `04a24f4e06`. **Deployment mode** Any deployment that uses a native Runner with Codex. ## What Changed - **Runner:** late usage no longer sets `reconciliation_required` or fails a run. For a turn that belongs to a previous run, the Runner emits `usage.reported` with the increment in `runDelta` and the owner in `reconciliation.priorOrigin`. `runDeltaAvailable` stays `false`, so an older Core does not bill it to the attached run. - **`CodexRunUsage::absorb_late_usage`:** treats only usage above the known thread total as late, and keeps the attached run's delta unchanged. A report at or below that total adds nothing, so the sum across runs always equals the thread total and a replay cannot charge twice. - **Delivery:** `captureLateUsageForOwner` stores the increment as a late receipt on the owning run. The owner must be a finished run in the same company and native session. If the Runner cannot name the owner, Core uses the session's last finished run and records the attribution as `inferred`. - **Local Runner:** late usage stays with the run that is active when it arrives, or the next run. Usage reported between two runs stays in the next run's delta, so the total stays exact. - **Accounting:** `reconcileLateRunUsageReceiptInTransaction` posts a token increase as a new cost event with the key `heartbeat:<run>:late:<receipt>`. It updates the run totals so the accounting integrity check still agrees, records `cost.late_usage_posted` in the activity log, and queues budget evaluation. - **No more refusal:** `persistUsageReceipt` no longer throws after it saves late evidence, so late usage cannot wedge a Runner outbox. - **Covered receipts:** an older same-source receipt at or below the accepted totals, with matching billing, is cleared because a newer cumulative receipt already covers it. Conflicting billing stays as debt. - **Older Runners:** a zero-filled invalidation stays on the delivering run. - **Unchanged:** a newer lower total, a changed model or billing type, and per-model receipts stay as visible accounting debt for an operator. - **Unchanged:** an unexpected provider message during the warm attachment drain still fails closed. A separate change addresses that gate. - **Docs:** added "Late provider usage" to `doc/execution-semantics.md`. ## Verification - `cd packages/paperclip-runner/runner && cargo test --locked -p paperclip-runner-core --test codex_provider`: 99 passed, 2 ignored. - `cargo test --locked -p paperclip-runner-core --lib codex_usage`: 7 passed. This includes new tests for the out-of-order and repeated report cases. - `cd server && pnpm exec vitest run src/services/run-cost-accounting.test.ts src/services/native-runtime/native-usage-delivery.integration.test.ts src/services/native-runtime/native-recovery-usage-delivery.test.ts src/__tests__/cost-data-quality.test.ts`: passed against embedded PostgreSQL before the last review fixes, and the accounting and delivery files were re-run after them. - `npx tsc --noEmit -p .` in `server` reports no errors. - New tests cover exact and inferred owners, repeated delivery, late usage that arrives before the owner's charge posts, another session's run, an older Runner's report with no increment, a same-run increase, and a lower total. They also run the accounting integrity check after posting. - **Review:** Greptile Apex scored 1/5, 4/5, 3/5, 2/5 and 4/5 while findings were fixed, then 5/5 at the current head. - **Local Runner:** driver tests cover usage reported between two runs, including across a restore. No end-to-end local run has exercised it. - **Not run locally:** the full test suite. CI provides that coverage. - **Not verified live:** a read-only check of a test deployment found no late receipts in its history, so this path has only run against fixtures. ## Risks - **Run totals can rise after a run finishes.** This is deliberate. The correction is a separate cost event and an activity log entry. - **Budgets are corrected after the fact.** A late charge can put an agent over budget after later work has started. Budget evaluation runs after each correction, but it cannot prevent that work. - **Local sessions accept imperfect attribution by design.** Their late usage can land on the active or next run, because local sessions have no durable path for routing it. This also applies after a Stop or a pause. The total stays exact. - **Out-of-order reports can leave tokens on the wrong run in the session.** If a prior turn's report arrives after the attached run has already counted past that total, those tokens stay on the attached run. Moving them would need a lower correction to the attached run. The total stays exact. - **Inferred attribution can name the wrong run in the session.** The usage still belongs to that session and the total stays correct. The attribution is recorded as `inferred`. - **A session with no finished run charges the delivering run,** as an inferred late charge after its own charge posts. - **Rollout order:** deploy Core before the Runner. An older Core would keep this event on the attached run as incomplete usage, and would still refuse a same-run late receipt. - **Migrations:** this change has no migration. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be reworked. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), through Claude Code, with tool use and code execution. A human directed the design and approved the scope. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Devin FoleyandPaperclip
|
c7390bef20 |
fix(server): stop one slow copy-back from starving workspace retries (#19111)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native run can accept its result before the workspace copy-back finishes. A background worker retries a failed copy-back from the retained sandbox without another provider turn. > - Until that retry succeeds, the run stays `running` and keeps its task's execution lock. User answers and wakes for that task stay queued. > - The worker checked one entry at a time (`batchSize: 1`, no concurrent checks). A slow sandbox export held every other task's due retry for the whole export. > - Work without a saved retry time used a deadline of `0`. Copy-backs that a restart interrupted always sorted ahead of retries that were due hours earlier. > - On a busy server that restarts often, scheduled retries never got to the front of the queue. Their tasks stayed stuck. > - This pull request runs up to four checks at once, one per task, and orders pending work by when it became pending. > - The benefit is that one slow export cannot stall unrelated tasks, and an old retry runs before newer work. ## Linked Issues or Issue Description Refs #17734 (added this worker), #18421 (the same concurrency pattern for native reply recovery) and #19063 (open; records why finalization failed a run). This pull request does not change their behavior. **What happened?** On a hosted instance, sandbox exports were slow for a few hours. A harness-state save timed out after about 5.5 minutes in 27 runs, and worker copy-back attempts took 2 to 34 minutes each. Eleven native runs then waited in `retryable_failure` (or `workspace_finalizing` with an expired lease) for more than three hours. Each one had exactly one copy-back attempt and no retry. The worker's operation receipts show its attempts ran strictly one after another. After each restart, the worker first took the copy-backs that the restart interrupted. They have no `nextAttemptAt`, so their deadline was `0`. The server restarted every 30 to 40 minutes, so the overdue retries never ran. One stuck run held a task whose user had answered a question card. The answer's wake stayed queued for the same three hours. **Expected behavior** A slow copy-back for one task does not delay due copy-backs for other tasks. Due work runs oldest first. New work without a saved retry time does not go ahead of older due retries. **Steps to reproduce** 1. Create two native runs with accepted results on different tasks. Put both in `retryable_failure` with an overdue `nextAttemptAt`. The first is older. 2. Make the copy-back for the older run block. 3. Start the worker. On `master` the second task's copy-back never runs while the first one blocks. The new worker test runs these steps against PostgreSQL. **Paperclip version or commit** `master` at `69dea878c0b`. **Deployment mode** Any server with native runs that use remote sandbox workspaces. ## What Changed - `server/src/services/native-runtime/native-workspace-finalization-work.ts`: - Remove `batchSize: 1`. Add `concurrentChecks: { limit: 4 }` keyed by `issue:<company>:<issue>`. Runs that share a task keep their copy-backs serialized. Ownership is still checked per run by the existing advisory lock and owner receipt. - Work without a saved retry time uses the coordinator's `updatedAt` as its deadline, not `0`. The value is clamped to now, because the database clock can be slightly ahead of the process clock. Saved retry times and lease expiry behave as before. - `doc/native-workspace-finalization-recovery.md`: one paragraph on worker concurrency and ordering. ## Verification - Failing first: "retries another task while an older copy-back is still running". On `master` the second task's copy-back never ran while the older one blocked (the test timed out after 15 s). Now the second task finalizes while the first is still blocked. After release, the first one also finalizes. - Failing first: "orders work without a saved retry behind older overdue retries". On `master` the interrupted copy-back had deadline `0` and sorted first. Now it sorts after the older overdue retry, and its deadline is not in the future. - `cd server && npx vitest run src/__tests__/native-workspace-finalization-work.test.ts src/__tests__/scoped-deadline-work.test.ts src/__tests__/server-startup-feedback-export.test.ts`: 3 files, 159 passed. After rebasing onto `69dea878c0b`, the first two files: 38 passed. - `cd server && npx tsc --noEmit -p tsconfig.json`: pass. - Not verified: the worker on a live instance under slow exports. The diagnosis comes from that instance's operation receipts and run events. ## Risks - Up to four copy-backs can now run at once on one server. Each one uses its own dedicated database connection and sandbox transfer. Live runs already finalize in parallel, so this is within the existing load pattern. - The batch size is now the default (25). A pass ends when its slowest check ends, but the other lanes keep processing the rest of the batch. - `updatedAt` changes on any coordinator write. A pending entry with new writes moves later in the order, but it stays due. - No schema migration. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) through Claude Code, with tool use (shell, file edits, test runs, read-only database and log queries). The runtime does not expose the context window or reasoning configuration. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
7d1e9b3477 |
fix(recovery): retain exact warm native export sources (#17916)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native task recovery must save unfinished workspace exports before continuation. > - A prepared export can still refer to a sandbox with a warm reuse policy. > - The strict recovery inspector rejects that policy and exhausts source verification. > - Ordinary reuse must not give the same unfinished allocation to another run. > - An existing user Retry can authorize bounded file maintenance after exact ownership checks. > - This change retains that exact source and preserves the original failure record. > - Recovery completes only when files, stop evidence and policy settlement agree. ## Linked Issues or Issue Description **What happened?** A failed native run has a prepared `keep_running` workspace export. Its exact sandbox lease still has `reuse_by_environment` policy. Native export cleanup can preserve that policy while it stops the sandbox. Generic workspace recovery then rejects the source with `lease_policy_not_retained`. A saved user Retry remains blocked before file export. **Expected behavior** Keep unfinished exports out of ordinary warm reuse. With current user authority and an unchanged saved Retry, retain the exact source for one bounded maintenance generation. Preserve files and return continuation to normal admission. Keep all ambiguous sources held. **Steps to reproduce** 1. Prepare a native workspace export with a warm sandbox and `keep_running` disposition. 2. Save a normal board Retry for the failed source run. 3. Persist its first exhausted source-verification failure before file export. 4. Run the normal maintenance worker. Before this change, source policy rejection leaves the saved files unrestored. Refs #17773 for source diagnostics. Related #17839 records a missing preservation obligation during blocked restart. This change handles an existing prepared warm export whose verification already failed. ## What Changed - Exclude unfinished physical allocations from reuse and obsolete-lease cleanup across every historical lease row. Reuse a clean candidate or acquire a fresh sandbox through normal selection. Preserve exact same-run reattachment, workspace-less execution and ordinary task admission checks. - Preserve the newer prepared remote-workspace writer proof for native and legacy runs. Share its binding check between single-source recovery and one batched allocation read. - Keep persisted recovery-action IDs when constructing continuation source checks. Synthetic sources cannot grant snapshot receipt authority. - Permit one audited maintenance adoption for the first exhausted source-verification attempt. Require an unadmitted canonical saved board Retry, current user authority, unchanged assignment and latest source run. - Keep all company, source, allocation, workspace writer, controller and physical finalization ownership checks. The public read-only inspector keeps its strict policy check. - Pin the exact source for file maintenance. Preserve the previous generation and reuse the existing bounded recovery worker. Create no provider turn or new user request. - Restore the warm policy in the same transaction as verified native copyback completion. A failed write or changed settlement keeps a recovery obligation. - When local merge metadata is missing, settle the exact verified snapshot while keeping its unfinished allocation retained. Deliver the saved Retry through normal fresh isolation and attach the verified files. Keep the accepted result as history. - Require native copyback proof before reopening an accepted result's original workspace finalizer. - Validate snapshot receipts through one shared boundary for warm, retained and ephemeral sources. Bind the receipt to the current generation's operation IDs, company, issue, run, recovery action, lease and provider allocation. - Retire fixture input before removing recovery proof, and use the same configured heartbeat for HTTP and worker dispatch. Keep the exact catalog refusal and no-provider-work assertions. - Document the recovery contract in technical documentation. Leave capability maps unchanged for this bug fix. ## Verification - Candidate-selection RED: all six clean-candidate and fresh-allocation controls fail on unchanged `f54e7baf538` because a pending export aborts the whole acquisition. GREEN: the focused suite passes **15/15 cases** after candidate filtering. Both provider paths preserve the protected lease and never resume or destroy its allocation. Obsolete fingerprints cannot send that allocation to cleanup. Historical-row fencing, modern native/legacy writers, exact same-run reuse, workspace-less execution and the single-read allocation-history check remain covered. - The existing reusable-runtime selection cases passed **40/40**. Driver/capability contract suites passed **5/5**. Reusable service controls passed **2/2**. After merging current master, the combined lease/provider suites pass 61 cases and the recovery integration selection passes 36 cases, including the upstream inherited-session and resolved-interruption assertions. This gives 97 passing targeted cases on the final source. A duplicate test-app declaration from the automatic merge was removed while retaining the upstream assertions. - The source-history repair has meaningful red/green evidence: three valid retired-handoff cases could not complete under the old blanket history-row rejection. Settlement now uses the same full source inspector as adoption. It accepts exact retired handoffs and still rejects a predecessor that becomes active after export. The 26 focused history, rollback, snapshot and ambiguity cases passed before this last merge. - Server typecheck passes on the final merged source. Module boundaries and `git diff --check` pass. Full local repository tests and a successful current-head full build are not claimed. Current-head CI is complete on `0ef974d76483e11fb0b0304bcb05bbd3b56e2ef9`: both required gates pass, with 54 successful checks, three expected skips, one neutral Paperclip review check and a successful security status. [Full CI evidence](https://github.com/paperclipai/paperclip/actions/runs/41904339370). Greptile reviewed that exact head at **5/5** with no actionable findings. Its check completed successfully. [Ponytail also reviewed the exact head at 5/5](https://github.com/paperclipai/paperclip/pull/17916#issuecomment-6640083408). No review threads remain unresolved, and the branch is mergeable. - Earlier local broad runs exposed a dropped recovery-action ID in the merged source projection and deferred test input being revisited after its records were removed. Both were repaired. Later local disk exhaustion caused manifest and saved-file reserve failures; generated caches were removed only from these checkouts. The affected saved-file control passed unchanged after cleanup. The original logs are retained. No disk reserve or timeout was lowered. - On the prior head, the first [full CI attempt](https://github.com/paperclipai/paperclip/actions/runs/41899038530) received simultaneous runner shutdowns. Four jobs were cancelled, and two late jobs have lost-runner annotations with unavailable logs. The Runner job had five earlier ACPX failures whose full assertion report was lost on interruption. Their cause remains unproven. Identical Runner source passed the [#19004 control](https://github.com/paperclipai/paperclip/actions/runs/41898305521/job/126814429725), including all seven enabled tests in that file. - That attempt's [browser shard](https://github.com/paperclipai/paperclip/actions/runs/41899038530/job/126817404595) returned to the exact saved task URL, but its conversation was still loading at the unchanged heading deadline. The [artifact](https://github.com/paperclipai/paperclip/actions/runs/41899038530/artifacts/13358840931) is retained. All five Cloud-auth cases passed locally on the unchanged head and in the [same-source CI control](https://github.com/paperclipai/paperclip/actions/runs/41898305521/job/126814429966). The delay remains unexplained. One rerun of failed/interrupted jobs was submitted after the workflow completed. This new source revision requires fresh CI. - The earlier [original-head workflow](https://github.com/paperclipai/paperclip/actions/runs/40552228175) passed all 48 jobs, and prior head `f54e7baf538` received Greptile 5/5. These results do not establish readiness for the new head. The earlier Superagent scan was neutral after a pinned-base checkout failure; its external analysis remains unverified. - These tests use real PostgreSQL and controlled provider boundaries. Hosted rollout and original-task acceptance remain unverified. ## Risks - Maintenance can resume and stop the exact sandbox for file export. It cannot admit a provider turn, choose another allocation or bypass normal continuation gates. - A saved Retry alone is insufficient. A changed source, newer run, active writer, competing lease, lost user authority or previous adoption keeps the hold. - Policy restoration can fail. Completion and restoration now commit together, so normal recovery remains responsible for a retry. - Snapshot preservation cannot repair missing local merge metadata. Its old allocation remains explicitly retained. A snapshot receipt grants no permission to reuse or destroy that allocation, or to replay the accepted result. - The allocation check uses one projected join per candidate over the existing provider lease index. It adds no polling timer or recurring scan. A fresh fallback can create another sandbox while the unfinished source remains retained; it grants no authority to execute a provider turn. - This change does not establish the cause of a prior provider failure or prove that any specific hosted sandbox still contains its original files. ## Model Used OpenAI Codex, GPT-6 family, for the original change. OpenAI GPT-6.1 Sol (`gpt-6.1-sol`), xhigh reasoning, for the merge and integration repairs. Both used repository tools and PostgreSQL verification. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Devin Foley <devin@paperclip.ing> |
||
|
Nicky LeachandClaude Opus 5.5
|
0173110ee3 |
fix(native): point failed run attachments at the docs (#19052)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - When a later run cannot attach to a native session, the run fails, and a later run may start a new provider thread. > - The first failure is easy to miss. In 5 of 11 cases on a test deployment, the run ended with a different error before it reported the attach failure. > - People and agents who diagnose a failed run look at its log and events, so that is where an explanation is useful. > - This pull request adds a pointer there at the first failure, with the real cause and a link to the docs. > - The benefit is that a failed attach explains itself, and readers can find the open follow-up in #19047. ## Linked Issues or Issue Description Refs #19047. I searched open and closed pull requests and issues for `prior_attach_failed` and related terms, and found no duplicate. ## What Changed - **Log:** `executeHeartbeatRuntime` writes one stderr line to the run log when a same-run retry follows a failed `run.attach`. - **Events:** `heartbeat.ts` adds a `lifecycle` event with the same message and the code `native_run_attach_failed`, using the stored redacted failure. - **Helper:** `describeFailedAttachment` extracts the real cause and returns nothing for other failures. - **Docs:** "Session settlement after a run" in `doc/execution-semantics.md` now explains this and links #19047. - **Link target:** the pointer links to the repo docs and not the issue. Agents read these logs, and anyone can comment on a public issue. - **Unchanged:** error codes and retry behavior. ## Verification - `cd server && pnpm exec vitest run src/services/heartbeat/runtime-execution.test.ts src/services/native-runtime/remote-warm-session-recovery.test.ts`: 57 passed. - `npx tsc --noEmit -p .` in `server` reports no errors. - New tests cover the log line at the throw site, and the helper's output for attach failures and other failures. - **Not covered:** the new event in `heartbeat.ts` has no test of its own. It uses the tested helper. - **Not verified:** I have not confirmed that agents read this log when they diagnose a failure. - **Not run locally:** the full test suite. CI provides that coverage. ## Risks - **Failure text in the log:** when the failure has no structured message, the log line includes the whole failure string. It passes through the run log's identity redaction. The event uses the stored failure that the executor already redacted. - **Link stability:** the link depends on the docs heading. Renaming that heading would break the anchor but not the page link. - **Migrations:** this change has no migration. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be reworked. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), through Claude Code, with tool use and code execution. A human directed the design and approved the scope. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
86afc2048b |
fix(native): admit follow-ups after an owner finished in this process (#19071)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native Runner per-turn sessions in a remote sandbox keep one conversation across task turns. > - Before a follow-up starts a new Runner, it must prove that the previous owner's controller stopped. #18846 added a retirement path for this proof. > - The native controller runs inside the webserver process. Thus the owner's recorded controller PID is the PID of the webserver. > - When the same server process serves the follow-up, that PID is alive and has the same start time. The proof reads this as "controller still alive" and refuses. The follow-up then fails with `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate`. This continues until a deploy replaces the process. > - This pull request uses the controller's per-process boot identity to recognize that this process recorded the owner. The process then uses its own session registry to prove that the owner's controller stopped. > - The benefit is that follow-ups on a long-lived server no longer fail before provider work. All other ownership fences stay as they are. ## Linked Issues or Issue Description Refs #18846 (retirement of recovered per-turn owners). Refs #19061 (names the refused retirement proof). The two changes are compatible. Whichever merges second needs a small rebase in `retireFinishedPerTurnRunnerdOwner`. **What happened?** On a test deployment, 17 runs over 14 tasks failed before provider work with `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate`. Each run had a per-turn lifecycle in a Daytona sandbox and got a new sandbox. In each inspected case, the previous run on the task succeeded and committed its result. Its run profile had no `nativeControllerRecovery` pointer. Without that pointer, `evaluateNativeControllerTakeover` is not lease-fenced, so it uses process liveness. The recorded `controllerPid` is `process.pid` of the webserver. When the follow-up is served in the same process, `process.kill(pid, 0)` succeeds and the start time matches. The result is `controller_still_alive`, and retirement refuses. **Expected behavior** A committed owner that this server process finished, released, and no longer runs is proven stopped. The follow-up then starts a new Runner, as #18846 intended. **Steps to reproduce** 1. Seed a per-turn sandbox owner that committed its result and released its lease. Record this process's controller identity (`currentNativeControllerIdentity()`) as the owner. 2. Keep the owner's file-backed journal in the local session root, and remove the database recovery pointer. 3. Admit a follow-up in a new sandbox. 4. Before this change, admission fails with `prior_owner_terminal_state_indeterminate`. **Paperclip version or commit** Observed on `e107a75755`. The same code is on `master`. ## What Changed - `native-restart-recovery.ts`: add `isCurrentNativeControllerProcess(owner)`. It is true only when the owner's `controllerBootId` is this process's per-start boot UUID and `controllerPid === process.pid`. A recycled PID or another container never matches. - `retireFinishedPerTurnRunnerdOwner`: when the owner was finished in this process and no `activeNativeSessions` or `nativeSessionStartups` entry exists for its run, treat its controller as stopped. Otherwise, use `evaluateNativeControllerTakeover` as before. The same in-process check runs again just before the archive. - All other retirement proofs are unchanged: committed result, released lease, no recovery state, settled journal, answered tool inputs, quarantine copies, and new or replacement sandbox. - `native-recovered-owner-followup.test.ts`: the seed helper accepts the owner's controller identity. New cases cover same-process admission (with and without a recovery pointer), an exited legacy controller, another live controller, and the same PID with a different boot identity. - `doc/execution-semantics.md`: describe the same-process stopped-controller proof. ## Verification - **Failure before the change:** with only the new tests applied, `cd server && npx vitest run src/services/native-runtime/native-recovered-owner-followup.test.ts` fails 1 case: "admits a follow-up in the process that finished the owner: local journal only". It expected admission and received `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate`, the exact error from the deployment. The safety cases already passed on the old code. - **After the change:** the same file passes 27 of 27. - Adjacent suites: `native-recovered-owner-followup`, `remote-warm-session-recovery`, `native-runner-restart-recovery.integration`, `native-restart-recovery`, `native-session-resume`: 210 passed, 20 skipped. - `native-session-executor.test.ts`: 1128 passed. - `tsc --noEmit -p server/tsconfig.json`: no errors. - **Not verified live:** this change has not run on a deployment. Run data came from read-only API calls only. No recovery was done on the deployment. ## Risks - **Second executor.** The new proof applies only when this exact process recorded the owner, and this process has no session or startup for that run. The database proofs still require a committed result and a released lease. Another live controller, or the same PID with a different boot identity, still holds the owner. Tests cover both cases. - **Scope.** The change applies only to the per-turn retirement path. The warm remote recovery path (`inspectRemoteWarmSessionRecovery`) is not changed. Warm handoff in the same process uses the in-memory idle owner instead. - **Hypothesis limit.** The deployment API does not expose finalization rows. Thus I inferred that the same process served each follow-up from the evidence: no recovery pointer, a local journal present, and no other failing proof. The local reproduction gives the exact error. #19061 can confirm the refused fence on the deployment. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic. Agentic coding in Claude Code with tool use (shell, browser for read-only staging API reads, code execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
61b2455319 |
fix(runtime): preserve failed sources and durable checkpoint authority (#19038)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A volume-free web server must preserve retained native session data. > - An old failed follow-up can keep its own snapshot while its controller file still belongs to the preceding successful turn. > - The Runner correctly refuses to import that file as the failed turn's execution authority. > - That refusal also blocks migration when no warm supervisor is present. > - This pull request preserves the file after it proves the older run's lineage, and uses existing typed database authority when present. > - Failed history and files survive, while normal execution authority stays unchanged. ## Linked Issues or Issue Description Refs #19024, which preserves the related older-supervisor case. The duplicate search found no PR for this older-controller case. **What happened?** Offline native-controller migration stops with `native_runner_checkpoint_binding_mismatch` for a failed retained follow-up. The local controller still contains the preceding successful run. A per-turn source can have no supervisor file, so the older-supervisor archive path does not apply. Read-only checks matched the database snapshots and controller-file hashes between the original source and the isolated rehearsal copy. **Expected behavior** Preserve the available file as evidence through the existing encrypted archive. Require a valid older controller, a completed successful prior run, and the same company, agent, task, native session, Runner and provider thread. Keep successful sources and active owners outside this archival fallback. When a valid typed database checkpoint already exists, derive only its missing recovery metadata and ignore obsolete local projections. **Steps to reproduce** Create a successful native turn and a later failed follow-up in the same session. Retain the successful controller file. Keep the failed snapshot and admitted input in the database. Leave the per-turn source without a supervisor. Run offline migration. The new positive regression fails with the binding mismatch before this change. ## What Changed - Share the scoped older-successful-run proof with the existing supervisor preservation path. - Validate the older controller through the Runner parser and artifact checks before archive publication. - Archive this failed legacy source without changing its snapshot, status, ownership, queued work or session pointer. - Preserve the normal refusal for foreign, changed, active, successful, invalid and already-durable cases. - Permit evidence archival for a stale observed coordinator only with exact host-owned stop receipts for every remote lease, no current task ownership, no result and no retry deadline. Recheck the proof before publication. - Use execution start for lifecycle ordering. A queued follow-up can predate its prior turn completion, and a remote stop can predate a later recovery timeout. - Resolve destruction receipts through the existing reusable-lease handoff verifier. Reject stopped tips, broken chains and competing tips. - Prefer a valid typed database checkpoint over a stale or absent local warm supervisor. Add only missing recovery metadata after current admitted-input and checkpoint binding checks. - Document the evidence and execution-authority boundary. ## Verification - Positive regression: red before the owner change and green afterward. - `pnpm exec vitest run server/src/services/native-runtime/native-controller-migration.integration.test.ts server/src/services/native-runtime/native-controller-recovery.integration.test.ts server/src/services/native-runtime/native-harness-backup-storage.integration.test.ts`: 81 checks passed across migration, recovery and durable storage. - The checks cover storage failure, repeated migration, exact file restoration after old-home deletion, unchanged history and ownership, and refusal to treat the archive as executable recovery authority. - A second regression reproduced the archive fence refusal for an observed failed source with a valid stopped-remote receipt. Negative controls cover missing, foreign, old and revoked receipts, live leases, pending results, retry deadlines and task ownership. Revoking the receipt during upload prevents publication. - Two additional regressions cover a follow-up queued before the prior turn finished and a remote stop before a later logical failure. Both fail with the earlier timestamp comparisons and pass with execution-start ordering. Genuine execution overlap and old receipts stay blocked. - Two historical source snapshots and controller-file hashes matched in a read-only live-versus-copy check. The first candidate reached the archive fence and exposed the observed-coordinator case. That failure is retained. A fresh candidate replay is required for the expanded proof. It does not qualify the full packaged migration or hosted Railway handoff. - Existing typed-authority checks cover stale and missing supervisors, repeat migration, byte-identical checkpoint rows, unchanged snapshots/owners/history, old-home deletion and ordinary queued-turn recovery. Foreign Runner, session and snapshot controls remain blocked. - Handoff checks cover an exact destroyed successor, a stopped successor, a broken proof and a competing tip. - Isolated replay of the two retained typed-checkpoint sources imported only derived recovery metadata. Checks confirmed unchanged checkpoint rows (12 and 9 parts), history, snapshots, finalization owners and task-session pointers. The replay used three source-module overlays with a blocked provider network; it does not qualify the released bundle. - Final-head CI attempt 1 passed typecheck, build and the migration suites but failed one unrelated budget-concurrency assertion in `decision-models.test.ts` (one provider call observed instead of two). The original log is retained. The isolated local comparison cannot classify the timing failure because this checkout has stale linked budget schema exports. The one retry of that failed CI job passed, and final-head CI attempt 2 is green. The original failure remains retained; its cause is not established. - Full local typecheck, repository tests and build are not claimed. The server typecheck reports pre-existing linked declaration mismatches outside the changed files; the full clean CI typecheck has passed. This checkout uses linked generated declarations; clean CI is required before merge. Prior head |
||
|
DottaandPaperclip
|
4740d251b7 |
fix(onboarding): retain plan approval across follow-up decisions (#19043)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Onboarding asks a person to approve a saved plan before execution tasks can start. > - A person can then approve a separate proposal for the first execution step. > - The creation gate checks only that latest card for the plan target. It loses the earlier approval of the unchanged plan. > - Decomposition and native context also omit checkbox plan confirmations. > - This pull request validates the latest execution decision and the latest plan decision together. > - The benefit is that approved work can start while human, revision, and proposal gates remain effective. ## Linked Issues or Issue Description Refs #17914, #18160, and #18283. The related-PR search found no duplicate repair. **What happened?** After a person accepted a checkbox plan card and a later execution confirmation, native task creation and REST child creation returned 403. Accepted-plan decomposition returned 422 because its lookup recognized only standard confirmations. Native task context returned no accepted plan for the checkbox decision. **Expected behavior** The latest execution proposal can use an independently approved, unchanged plan. A pending or rejected latest proposal must still stop execution. The current plan must retain valid human approval of its exact revision. **Steps to reproduce** 1. Save a plan on a server-created onboarding task. 2. Create a revision-bound checkbox confirmation and accept it as a human. 3. Create and accept a separate, untargeted confirmation for the first execution task. 4. Create the child through the native tool or REST API. 5. Separately, try accepted-plan decomposition or read native task context after step 2. **Paperclip version or commit** The workspace's registered hosted release is Core `09c05c55050fede448196637cbaa543a154da76e`. The three regressions also failed against local source `3cdb367369f29692b81cf19b514a2abdba90734f`. **Deployment mode** Hosted native Runner with ACPX Claude. Local regression proof uses actual PostgreSQL, authenticated HTTP routes, native tool authority, and synthetic task data on macOS arm64. No customer data or credentials are in the fixtures. ## What Changed - Validate the newest ordinary proposal and the newest plan proposal separately. Keep the existing human-card and conversational provenance checks for each. - Require the accepted plan target to match the current document and revision. A newer pending, rejected, invalid, or foreign plan decision cannot fall back to an older approval. - Share persisted confirmation parsing across admission, decomposition, and native context. Validate checkbox selections against the saved options and limits. - Return `acceptedPlan.approvedOptions` with the user's saved selected IDs and option text on later wakes, including revision-pinned context. Keep unchecked defaults out of scope. - Add behavioral and safety regressions. Document the approval rule. ## Verification - Three behavioral regressions failed before the repair: child creation returned 403, decomposition returned 422, and native context returned `acceptedPlan: null`. All three now pass. - Two scope-retention regressions failed before the review fix: native context omitted the selected option after a later approval, with and without a pinned plan revision. Both now pass. Unchecked defaults stay excluded. - Current admission, native-tool, and accepted-confirmation replay suites: 131 tests passed. Existing first-task completion safety checks: 17 passed, including the atomic decision recheck. - Safety cases cover missing and rejected plan approval, agent-only decisions, foreign source runs, changed revisions, invalid checkbox selections, newer pending or rejected proposals, and task-creation replay. - Full repository `pnpm -r typecheck` and `pnpm build` passed locally on final head `7c0e8ab784`. - [Full final-head CI](https://github.com/paperclipai/paperclip/actions/runs/41911580531) passed: all Vitest shards, Runner checks, eight browser E2E shards, build, release registry, and canary dry run. The required `verify` and `e2e` aggregates passed. Storybook checks were skipped for this backend change; the file-selection job was skipped because full CI ran. - Earlier local typecheck attempts exposed Copilot verification during build preparation, stale declarations across the rebase, and two test-fixture typing errors. The assets were verified, declarations rebuilt, and fixtures corrected before the complete final-head pass. - Full local Vitest attempts were interrupted for the rebase and review repair. The duplicate final local run was stopped after full Linux CI passed. No complete macOS Vitest-suite pass is claimed. - A fresh Claude-provider acceptance run remains unverified: the local Claude account is signed out and no Anthropic API credential is available. The actual native tool and HTTP paths are verified with real PostgreSQL persistence. This PR does not deploy or recover the reported production task. - Greptile: 5/5 on `7c0e8ab784`, with the review thread resolved. ## Risks A later acceptance must not revive rejected or changed scope. Admission validates both current decisions and retains source, company, human provenance, and document locks. The extra plan-proposal lookup is bounded and limited to accepted, untargeted first-task proposals. Ordinary delegation adds no new proof lookup. The native context adds an optional `approvedOptions` projection. Revision-pinned context now performs the existing scoped approval read to recover that selection. No database schema, request, provider, or background worker changes are included. ## Model Used OpenAI Codex, exact model `gpt-6.1-sol`, max reasoning effort, with repository, shell, browser, and GitHub tools. The current session metadata confirms the model and effort. The context window size is not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
8a0e7ab511 |
fix(runner): preserve ordinary append for unscoped governed terminals (#19044)
## Thinking Path > - Paperclip manages work done by AI agents. > - A native turn can park on a saved human question or approval. > - Its terminal event can return a committed receipt before observers finish. > - That option requires an explicit turn ID at the event port. > - The protocol also permits events without a turn ID. > - The runner requested the option for those events, which the port rejected. > - This change keeps those events on the ordinary append path. ## Linked Issues or Issue Description Refs: #19025. **What happened?** The native governed-wait observation can accept a schema-valid `item.completed` without a turn ID. If the later terminal also has no turn ID, the runner compares two absent IDs and requests `deferPostCommitObservers`. The event port rejects that option. The run then fails with `governed_wait_provider_terminal_unproven`. **Expected behavior** Only a terminal with an explicit turn ID can request the early receipt. An unscoped terminal must keep the ordinary append behavior and await its observers. The event port must retain its turn identity check. **Steps to reproduce** 1. Create a pending human-only interaction from the run. 2. Pass a schema-valid completed tool event without a turn ID through the real governed-wait observation. 3. Supply an interrupted terminal without a turn ID during the cancellation drain. 4. Run the actual native runtime with the PostgreSQL event port. The parent code rejects the deferral request and reports a failed terminal. **Paperclip version or commit** Reproduced on `4ea53f9eb5a24de10db3dfda18456132987c4af8`. The follow-up was identified by a review published after #19025 merged. No observed provider emission or original-task recovery is claimed. I checked current master and related PRs before this correction. **Deployment mode** Built from source. The regression uses real PostgreSQL and a deterministic provider boundary; it makes no model call. ## What Changed - Require a truthy terminal turn ID before the runner requests deferred observers. - Keep ordinary append behavior, the port identity check and all timeouts unchanged. - Exercise the production governed-wait observation with a persisted pending interaction and the real event port. - Retain exact-turn observer and checkpoint controls. Add a direct rejection control for unscoped deferral. - Document that unscoped terminals await their observers. ## Verification - First meaningful red: the actual observation, native runtime and PostgreSQL port fail on the unmodified parent runtime. The new case reports `NativeProviderTerminalFailure` in 143ms; this is not a setup or deadline failure. - Focused regression and authority controls: 12 passed, 69 filtered. Full runtime suite: 203 passed. Full PostgreSQL port and timing suites: 84 passed. - The tests leave the human-only interaction pending with continuation policy `none`, accept one result, and start only one turn. Existing scope, invalid evidence, checkpoint, abort, duplicate and restart controls remain. - Exact candidate: `8840ea2c9c308e5a60423e67fb3612d24029ba84`. Repository-wide `pnpm -r typecheck`, `pnpm build` and `pnpm check:module-boundaries` passed on this exact head. `pnpm test:run` is still in progress and is not reported as passing. Current-head CI passed: 55 success, 3 skipped and 1 neutral, including required `ci / verify` and `ci / e2e`. Greptile and Ponytail both reviewed this exact head at 5/5 with no blocking findings. The native review was source-only. No unresolved review threads remain. - No UI, database schema, agent permission or mapped user entry point changes. No deployment, live retry or provider replay was performed. ## Risks Unscoped terminals retain the existing observer wait and can still fail if it exceeds the existing drain deadline. They do not gain an early receipt. The change does not supply a missing turn ID or weaken source, company, run, session, cancellation or transaction checks. A PRP terminal is not physical-stop or accounting-completion proof. ## Model Used OpenAI `gpt-6.1-sol`, with code and shell tools. The host does not expose the reasoning mode or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
22735cfcea |
fix(native): preserve governed terminal receipts during observer settlement (#19025)
## Thinking Path > - Paperclip manages work done by AI agents. > - A native turn can yield to a saved human question or approval. > - The provider must stop, and its terminal event must be saved. > - The old drain waits for both the event transaction and its later observers. > - A slow observer can make a saved terminal look unproved. > - This change separates the committed receipt from owned observer work. > - Checkpoint and result acceptance still wait for that work to settle. ## Linked Issues or Issue Description **What happened?** A native governed wait can fail with `governed_wait_provider_terminal_unproven` after its exact interrupted terminal is already in PostgreSQL. The event port waits for postcommit accounting, output and projection callbacks before it returns the receipt. Those callbacks can cross the provider drain deadline. **What did you expect to happen?** A committed terminal proves the event transaction. Its observers remain separate work. A checkpoint or result must not be accepted until those observers settle. A pending human-only question must keep its audience and continuation policy. **Steps to reproduce** Run the real PostgreSQL control-plane regression. Commit a same-turn interrupted terminal. Hold its postcommit observer beyond the existing provider drain. The parent code reports an unproved provider terminal. The fixed code retains the committed receipt and joins the observer before checkpoint or result acceptance. No overlapping repair was found in the current master or open-PR search. Related [#18977](https://github.com/paperclipai/paperclip/pull/18977) covers final conversation-response selection. It does not change this receipt boundary. The source reproduction is consistent with the observed failure. The exact hosted provider code is unavailable, so it does not prove that every observed failure used this branch. ## What Changed - Add an explicit terminal-only option to the internal event port. Ordinary appends keep their existing behavior. - Return that receipt only after the normal transaction and company, agent, run, session, source and revocation checks. - Retain the full observer promise. Join it before checkpoint and result writes. Keep its original rejection and observe it after an aborted join. - Reuse the existing provider drain and checkpoint deadlines. Preserve duplicate and restart replay without another model turn. - Document the receipt and duplicate-observer replay boundaries. Keep human-only and no-continuation cards unchanged. - Assert the actual Codex facade requests deferral only for its interrupted terminal. Isolate empty skill inventory in monitor-only process fixtures while retaining their real scheduler, lifecycle, selection snapshot and three-second deadline. ## Verification - Meaningful parent red: actual native runtime and real PostgreSQL event port. The terminal row commits, but a held observer causes `NativeProviderTerminalFailure` with `governed_wait_provider_terminal_unproven`. - Current candidate `0db24ae74ee42fdf92035d8407fc095e9f73cc6a`: full runtime suite, 203 passed. Real PostgreSQL control-plane, timing and monitor suites, 103 passed. Runner TypeScript and full layered server typechecks passed on the unchanged candidate source tree. The controls cover uncommitted and foreign evidence, explicit false and failed terminals, held and rejecting observers, abort and controller revocation, duplicate receipts, and new-port recovery. The real pending human-only card stays unanswered with continuation policy `none`. Recovery does not call `startTurn` again. - `pnpm -r typecheck`, the production server build (including the real Runner release binary), module boundaries and vendor dependency verification passed on `6d75`; the follow-up changes only tests and documentation. Diff checks pass on the current candidate. The prior `399ad3dc2b` repository-wide `test:run` was stopped as obsolete during the general-server phase after preserving its progress (exit 130, incomplete); its repository build was not reached. Its six completed chat shards passed 1,904 cases, but this is prior-head evidence and not a full-suite pass. Full current-head CI is required before merge. - CI attempt 1 on `6d75` failed one real three-second monitor-settlement assertion; the aggregate verify gate consequently failed. Other jobs passed. The actual CI merge tree was `3037bf6db162da3b0a9172046360401f8449ebc5` (`6d75` into `89703c3`). An unchanged full-file replay reproduced three timeouts. Original-deadline inspection found the legacy process still preparing; local phase timing measured cold skill inventory at 6.629 seconds. The original CI did not retain that phase detail, so its exact subcause remains unproved. The fixture correction supplies empty skill entries for these skill-free process tests and still invokes the selection-snapshot callback. It preserves all assertions and the three-second deadline, with no new drain wait. All 21 cases pass; deliberately restoring the old blocked-status rejection still fails the real blocked-check case with 409. No CI retry was issued. - The prior current-head native review was 4/5 with no production correctness finding. Its requested direct facade-options assertion and duplicate-observer wording correction are included. Current-head CI is green (55 success, 3 skipped), including required verify and E2E; current-head Greptile is 5/5 with no actionable findings. The old native advisory remains tied to `6d75`, and its requested findings are addressed. No new native assessment is claimed or required by protection. - The first draft broke three existing transaction-start timing controls. The no-debt path now keeps the original synchronous start, and all three pass. Two draft type errors were corrected. A further real-PG red found an extra default-append assertion after commit. Removing it restores the original committed receipt after a late abort; the same aborted signal still cannot write a checkpoint or result. Original setup and assertion failures remain in private verification receipts. - Normal merge of current master `3a2249dd5768313f04b2aa3860dfd363492eda16` has no conflicts. The affected production source blobs did not change in that baseline merge. - Local proof uses Node 24.22.2 on Darwin arm64, a real PostgreSQL instance and deterministic provider-boundary fixtures. It makes no model call. No deployed replay or original-task recovery is claimed. ## Risks Deferred observers must remain owned. A missed join could accept incomplete accounting or output. A rejected or stalled observer must prevent acceptance, and an abort must prevent late checkpoint/result writes. The tests exercise these boundaries. The port keeps the existing best-effort trace-publication handling. This change does not turn a PRP terminal into physical-stop or accounting-completion proof. ## Model Used OpenAI `gpt-6.1-sol`, with code and shell tools. The host does not expose the reasoning mode or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Nicky LeachandClaude Opus 5.5
|
e107a75755 |
fix(native): decide session settlement from live state (#18992)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native Runner sessions keep one provider conversation across task turns, so a later run attaches to a session that an earlier run settled. > - Three gates decide if a session is settled: Core's journal check, the Runner's attach check, and the Runner's drain of provider messages after a turn. > - Each gate kept a list of harmless events and refused anything else. Every new harmless event became an outage. > - An example: Codex replays a settled turn's usage on `thread/resume`, so `run.attach` refused every session that restarted after its idle timeout. > - Events after a terminal describe the past. Refusing the session cannot undo them, and it hides them. > - This pull request asks a different question: is anything live or waiting right now? Past events are recorded and no longer block. > - The benefit is that harmless events stop breaking session continuity, with no event list for Core and the Runner to keep in step. ## Linked Issues or Issue Description No issue exists. I searched open and closed pull requests and issues for `run.attach requires a settled Codex provider session` and `prior_owner_terminal_state_indeterminate`. Related work: #18890 added one more accepted event to Core's list. This change replaces that list. #18991 handles late usage and is independent of this change. **What happened?** On a test deployment, 11 runs failed in 21 days when `run.attach` refused with `requires a settled Codex provider session with no pending events`. In all 11, the four pending events were two startup audit events, `session.resumed`, and a `codex_resume_usage_snapshot` diagnostic. The diagnostic is what the attach check refused. The retry then failed with `prior_owner_terminal_state_indeterminate` in 6 of them, which hid the first failure. **Expected behavior** A session with nothing live or waiting accepts the next run. If attachment fails, the run reports the real cause. **Steps to reproduce** 1. Finish a Codex turn on a native Runner session. 2. Wait for the idle timeout so that the provider process stops. 3. Start the next run. Codex replays the usage for the settled turn on `thread/resume`. 4. The attach fails. `durable_backend_attaches_after_a_settled_resume_usage_snapshot` reproduces this on the unfixed code. **Paperclip version or commit** Observed on `8fd917aedf`. The same check is on `master` before this change. **Deployment mode** Cloud deployment, with a native Runner in a plugin sandbox environment. ## What Changed - **Runner attach:** `attach_run` refuses only for live or waiting state: no thread, a closed session, a session that needs reconciliation, an active or ambiguous turn, an active goal, a pending tool call, or an unanswered provider request. The error now names the reason. - **Pending events:** they no longer block attachment. They stay in the retained queue, and the Runner commits them under the prior authority before it rotates. `session.resumed` is still discarded because it is superseded. - **Runner drain:** `drain_completed_turn_tail_for_warm_attachment` no longer needs a list of known messages. It still fails on a message that names another turn, on `turn/started`, on a provider request, and on a process exit. - **Core:** `isSettledRemoteWarmControlState` accepts any authenticated event after the terminal unless it starts work (`turn.started`, `turn.accepted`, `run.attached`, `tool.execution.started`, `semantic_tool.input`, `runtime_request.created`), or a goal snapshot or update shows an active goal or work in progress. All identity, source and protocol checks are unchanged. - **Visibility:** the next run's log names the event types that were recorded after the previous run finished. - **Real cause:** a same-run retry after a failed attachment now fails with `prior_attach_failed: <original failure>`, instead of `prior_owner_terminal_state_indeterminate`. - **Docs:** added "Session settlement after a run" to `doc/execution-semantics.md`. ## Verification - `cd packages/paperclip-runner/runner && cargo test --locked -p paperclip-runner-core --test codex_provider`: 102 passed, 2 ignored. `--lib`: 478 passed, 2 ignored. `--test durable_recovery`: 3 passed. - **Failure before the fix:** `durable_backend_attaches_after_a_settled_resume_usage_snapshot` failed on the unfixed code with the same error as production. - `cd server && pnpm exec vitest run src/services/native-runtime/remote-warm-session-recovery.test.ts src/services/native-runtime/native-session-executor.test.ts -t "settled remote warm control evidence|recovers a blocked|automatically recovers only a proven settled local session|retained|warm"`: 476 passed, 680 skipped by the filter. - `pnpm exec vitest run src/services/native-runtime/native-session-resume.test.ts`: 57 passed. - `npx tsc --noEmit -p .` in `server` reports no errors. - **Real data, read-only:** I decrypted the 11 failed runs' stored events on a test deployment and printed only event types, field names, and thread and turn IDs. All 11 show the same four events. - **Audit of live state:** I read the Runner's provider state and request tracking to build the attach check. It is a code audit. I have not proved that every kind of live work sets a flag. The Runner holds active turns, ambiguous starts, tool calls and provider requests in state today. - **Not covered:** the wiring that throws `prior_attach_failed` has unit coverage for its message only, not an end-to-end test. - **Review:** Greptile Apex scored 4/5, then 5/5 at the current head. - **Not run locally:** the full test suite. CI provides that coverage. - **Not verified live:** no live run has used this change. ## Risks - **Fail-open for unknown live work:** an event type that starts work, but is not in the short list and sets no state flag, would now pass. This is the main tradeoff. The previous list failed closed, but it caused real outages. - **More events reach the journal after a terminal.** The usage replay diagnostic is now delivered rather than refused. These events are informational, and they are named in the next run's log. - **A failed attachment is not retried on the same run.** Some refusals could clear on their own, such as a turn that finishes, and some attach errors are not about live work, such as a failed provider restore. The run now reports the cause, and it needs another run to continue. On a test deployment, the next successful run after each failure used a new provider thread and a new execution workspace. A later change could add same-run retry. - **Rollout order:** deploy Core before the Runner. An older Core would refuse the new events that a new Runner commits after a terminal. - **Migrations:** this change has no migration. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be reworked. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), through Claude Code, with tool use and code execution. A human directed the design and approved the scope. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
3a283fbc9b |
fix(runtime): preserve failed sources with older warm supervisors (#19024)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A volume-free web server needs durable recovery data for retained sessions. > - An old failed follow-up can retain its snapshot while the warm supervisor still holds the preceding successful turn. > - The migration refuses to treat those different snapshots as the same execution authority. > - That refusal also blocks the tenant even when the failed source can be preserved as an archive. > - This pull request archives that narrow case after it verifies the older successful snapshot and all scope bindings. > - The failed run keeps its history and files while valid controller recovery keeps its existing proof requirements. ## Linked Issues or Issue Description Refs #18959, which added the existing failed-controller archive path. No duplicate PR for this case was found. **What happened?** Offline controller migration stops with `stale_supervisor_snapshot` for an old failed warm follow-up. Its local supervisor still contains the prior successful turn. Read-only checks found the same database snapshot and supervisor bytes in the live source and the isolated copy. The mismatch predates the rehearsal. **Expected behavior** Preserve the failed run's database state and available local files in the existing durable archive. Only archive when the older supervisor matches a completed successful run in the same company, agent, task, native session and Runner instance. Keep current execution and controller authority checks intact. **Steps to reproduce** Create a successful native turn and a later failed warm follow-up. Keep the successful snapshot in the supervisor file and the failed snapshot in the database. Give the failed source local controller bytes but no durable controller checkpoint. Run the offline migration. The new regression fails with `stale_supervisor_snapshot` before this change. ## What Changed - Verify the older supervisor against scoped successful history and the same provider thread. - Use the existing encrypted archive owner for that failed legacy source. - Preserve snapshots, run status, ownership, queued work and the session's last-run pointer. - Keep foreign, changed, active, successful and already-durable sources outside this path. - Document the preservation and recovery boundary. ## Verification - New positive regression: red on the original owner, green after the change. - Native migration and controller recovery: 27 checks passed. Final migration checks also cover refusal to replace a durable checkpoint. - Storage failure blocks publication. Repeated migration is safe. The archive restores the exact controller and supervisor bytes after the old home is deleted. - The archived failed source does not gain executable recovery authority. - Local server typecheck is blocked by reused workspace declarations for accounting, bootstrap and ingress contracts. No diagnostic names either modified owner or test file. Full local typecheck, repo-wide tests and build are not claimed; clean CI is required before merge. The exact-head CI has now passed typecheck, build, policy, Runner checks, browser E2E, and all server test shards. - Rehearsal verification against the copied historical source remains required. Full tenant migration and hosted deployment qualification remain separate gates. ## Risks The archive preserves evidence for an already failed source. It does not certify that source as resumable. A prior snapshot must match exactly, and the normal archive owner checks the terminal failed binding before it publishes. Existing durable checkpoints remain authoritative. A missing or changed proof still blocks migration. ## Model Used OpenAI Codex, exact model ID `gpt-6.1-sol`, with reasoning, tool use and code execution. The host does not expose context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in-PR following the bug template - [x] I have not referenced internal or instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal ticket id or instance-derived details - [x] I have run the relevant tests locally; retained failures and limits are listed above - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
cf0ab4f7f4 |
chore: enable boxd plugin release publishing (#19020)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Sandbox provider plugins let those agents run in remote environments. > - The boxd provider exists, but its release entry excludes it from CI publishing. > - npm now has the `@paperclipai/plugin-boxd@0.0.0` bootstrap placeholder. > - This pull request enables CI publishing and documents npm installation. > - A pre-existing OpenCode fixture blocked CI because it qualified a group-writable host Node binary. > - This also gives that test its own Linux Node copy with strict permissions. > - Operators can install a real release after CI publishes it. ## Linked Issues or Issue Description Related implementation: https://github.com/paperclipai/paperclip/pull/17284. No matching public issue or duplicate release-enrollment PR was found. **What existing behavior does this improve?** Publishing and installing the existing boxd sandbox provider. **Subsystem affected** `packages/plugins` and the npm release manifest. **Current behavior** `publishFromCi` is false. npm has only the `0.0.0` placeholder. The README says the package is not published and requires a local-path install. **Proposed behavior** Include boxd in CI releases. Document stable installation, explicit published versions, and the placeholder limit. **Reason and benefit** Operators can install the provider through the existing npm plugin installer after a real release ships. **Breaking changes** None. This changes release enrollment and documentation only. **Additional context** This follows the existing plugin and sandbox roadmap items. It adds no new runtime capability. ## What Changed - Set `publishFromCi: true` for `@paperclipai/plugin-boxd` in the release manifest. - Replace the README's unpublished-package notice with npm installation guidance. - Explain that the `0.0.0` placeholder has no plugin functionality and that `latest` can point to it until the first stable release. - Keep the local development instructions available. - Fix the existing ingress-retry test to use an owned Linux Node copy and explicit executable permissions. This follows the existing OpenCode transport fixture. It does not change production launch checks. ## Verification - Passed: `node --test scripts/release-package-map.test.mjs scripts/check-release-package-bootstrap.test.mjs scripts/bootstrap-npm-package.test.mjs` (31 tests). - Passed: `node scripts/release-package-map.mjs check`. - Passed: `PAPERCLIP_RELEASE_BOOTSTRAP_BASE_SHA=origin/master node scripts/check-release-package-bootstrap.mjs scripts/release-package-manifest.json packages/plugins/sandbox-providers/boxd/README.md`. - Passed: `git diff --check`. - Confirmed that npm resolves `@paperclipai/plugin-boxd` to version `0.0.0`. - Full workspace typecheck, test suite, and build were not run locally. CI checks remain required before merge. - Initial CI failure: [native Runner job](https://github.com/paperclipai/paperclip/actions/runs/41889969053/job/126785636239), OpenCode ingress retry, `qualified OpenCode proxy command must not be group- or world-writable`. The failing fixture is byte-identical to the base commit `3a2249dd576`. It passed `process.execPath` to qualification, while other OpenCode tests already use a private Linux Node copy. - Local focused run: `pnpm exec vitest run server/src/services/native-runtime/native-ingress-bootstrap-retry.integration.test.ts` built the real Rust Runner and passed the Codex case. The OpenCode case stopped at `runner_remote_provider_artifact_incompatible: OpenCode proxy path belongs to the controller host` on macOS (`/Users/...`). This does not exercise the new Linux-only copy. - An unchanged-base OpenCode comparison was attempted with `-t "reacquires opencode ingress"`, but PostgreSQL setup failed with `ENOSPC` before the case ran. The local baseline comparison is therefore inconclusive. Task-owned Rust build outputs were cleaned to recover disk space. - At this stage Linux verification was pending. The successful final run is recorded below. - No new tests were needed. The existing release-tool tests cover manifest enrollment, dependency ordering, and bootstrap checks. - Current-head CI interruption: [workspaces-a shard 1/2](https://github.com/paperclipai/paperclip/actions/runs/41893808561/job/126799012853) was cancelled after the runner received a shutdown signal. The log shows no assertion failure. The job rerun request was rejected with HTTP 403 because the workflow was still running. The subsequent fixture correction starts a new full CI run and supersedes that pending rerun. - CI at `7fffa315`: [native Runner job](https://github.com/paperclipai/paperclip/actions/runs/41893808561/job/126799011711) passed executable permission checks, then failed because Node tried to resolve `/memfd:paperclip-verified-launch (deleted)`. The private executable was named `opencode-node`, which bypassed Runner's Node interpreter recognition. The fixture now names its copy `node`, matching the existing transport fixture and selecting the CommonJS descriptor loader. - Passed a local smoke test of the production CommonJS descriptor loader using an open, unlinked script file. This is macOS evidence only; the successful final Linux run below provides the complete ingress retry evidence. - Final verification on `bcacac542aa0a1b641b26c099c1e9fc4a1653ec4`: all 54 reported checks passed; three unrelated checks were skipped. [Native Runner integration job](https://github.com/paperclipai/paperclip/actions/runs/41895787712/job/126805844773) passed, including the OpenCode ingress retry regression. The fresh full run also passed the workspace shard that was interrupted on the previous attempt. Greptile reports 5/5 for this exact head with no unresolved review threads. Earlier failures and local verification limits remain recorded above. ## Risks - The maintainer confirmed that npm trusted publishing is configured for this package. The first real CI publish will verify OIDC authentication. - Merging enables publication in the next successful canary run. An installation that uses `latest` can still resolve to the placeholder until the first stable release includes this change. ## Model Used OpenAI Codex, exact model ID `gpt-6.1-sol`, with xhigh reasoning effort, repository tool use, and shell execution. The runtime does not expose the context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
3a2249dd57 |
fix(runner): retry proven ingress bootstrap failures (#18981)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native execution must retain its admitted model and execution authority during recovery. > - A sandbox provider can reject Runner endpoint acquisition before the Runner gets a credential. > - The old error path loses the structured HTTP status and leaves that startup failure ambiguous. > - A zero counter in an old checkpoint cannot prove that no credential was issued. > - This pull request adds a versioned producer proof and carries closed provider status through the existing RPC. > - A proved transient startup failure uses the existing retry budget. A terminal proved-unused run can settle its accounting. ## Linked Issues or Issue Description Refs #18018 and #18934. Those changes restore existing checkpoints and ready retained environments. This change covers endpoint acquisition before Runner dispatch. A search of related public PRs found no duplicate implementation. **What happened?** A provider HTTP 502 during Runner endpoint acquisition becomes an untyped worker error. Native recovery treats it as ambiguous and stops without retrying. The run's native usage receipt also stays pending after startup ends without model work. **Expected behavior** Retry a structured transient endpoint failure only when the complete current controller checkpoint proves that this exact run never issued a Runner credential. Preserve authentication, configuration, Stop, ownership and provider-progress gates. Settle zero usage only under the existing terminal accounting owner. **Steps to reproduce** 1. Use the controlled endpoint in the Daytona ingress integration test to return HTTP 502. 2. Register the actual native Runner driver and persist its bootstrap journal through the encrypted checkpoint writer. 3. Observe the old terminal ambiguous disposition instead of bounded startup recovery. **Paperclip version or commit** Reproduced on the pre-change master source. The original regression used the production native driver, a real HTTP fault, a real durable control plane and local Postgres. The new contract additionally tests the pinned Daytona SDK and real worker RPC. ## What Changed - The controller owns a v2 persisted state with retained origin and credential history. Current readers still accept v1. Older writers reject v2. Restore and checkpoint decoding do not add missing history. - The proof validates the complete bootstrap journal, all six identity fields and every delivery, credential and progress control. The server rechecks its fingerprint under the current controller claim before retry or accounting changes. - Daytona and the SDK carry a closed acquisition phase and HTTP status through the existing ingress RPC. Both public error wrappers omit response bodies and causes, including Pino cause serialization. - Only structured HTTP 429 or 5xx ingress failures can use the existing three-attempt startup budget. HTTP 401/403, configuration and unknown errors do not gain retry authority. - A terminal proved-unused run wakes existing accounting and records zero provider work. It does not create a semantic result, prove physical termination or release an unknown process/workspace hold. - The normal automatic second attempt rechecks its exact claim, frozen execution and matching complete local/encrypted journal before admitting absent remote Runner state. It keeps the original pending bootstrap command IDs. A proved unused Codex home can rematerialize sealed skills. Full raw local state is checked before streaming-event or policy projection. - Normal saved-state readers and the existing E2E journal evidence matcher accept the closed v1/v2 set. The separate legacy no-launch witness is unchanged. Technical docs describe the contracts. ## Verification Current head: `cd38f70409cdb8664e14bfbf4efa208f468a0419` (final delta adds only fixture artifact preparation). Production files are unchanged from the independently reviewed `93c8a33520` source. Full repository typecheck and build passed at `96417acb58`; the affected server typecheck also passes after the final two-line fixture delta. Focused protocol, raw-proof, real retry, SDK and RPC acceptance pass on this source. Earlier broad suite counts are listed with their limits below: - Durable Runner protocol: 187 tests. - Native executor, driver, startup/accounting and redaction: 1,211 tests, including real Postgres and encrypted checkpoint persistence. - Normal saved-state restore/projection compatibility: 21 focused cases. - SDK: 209 tests. - Standalone Daytona: 544 tests; 8 opt-in live tests skipped. - Explicit standalone `pnpm -C packages/plugins/sandbox-providers/daytona exec vitest run --config vitest.sdk-ingress.config.ts`: 6 cases with real pinned SDK 0.203.0 and worker RPC ingress acceptance. These include lookup 502/401/403/400, private-preview 502 after successful lookup/refresh, missing credentials and private-response exclusion. - E2E journal evidence: 4 cases across both supported formats, with wrong ownership, missing stimulus and unknown-schema controls. - The exact historical controller source rejects a new v2 checkpoint at decode and restore. A v1 control is accepted. This verifies the downgrade fence with the actual older parser. - Startup controls cover issued and pruned credentials, old/truncated journals, provider progress, exhausted retry accounting, and checkpoint/Stop/controller-claim races. Race cases retain the current execution and pending usage. - New exact admission controls: 19 cases through encrypted persistence and cold restore. They cover current/stale/expired claims, Stop, pause, model/trust/account changes, prior progress, issued/pruned credentials, adopted/old/truncated state, local/canonical fingerprint mismatch, hidden delivery, unsupported work and company scope. Two additional red/green cases prove that raw local stream progress and missing event policy are rejected before lossy checkpoint projection. No case mutates run, claim, result or accounting state. - Actual native executor, scheduler and Rust Runner retry: 2 cases. Codex uses its fake app server. OpenCode uses the real proxy and fake OpenCode HTTP server with the frozen DeepSeek model and allow permission configuration. Each path launches no Runner on attempt 1, starts one Runner and provider on attempt 2, reuses both original command IDs, commits one accepted result, validates the persisted local workspace, records nonzero synthetic native usage and settles accounting once. No zero-work accounting flag survives. The controlled SDK and protocol tests use local source, isolated PostgreSQL, a private HTTPS preview and fake provider servers on macOS arm64 with Node 24.22.2. They create no hosted sandbox and make no model call. The root workspace intentionally omits standalone provider dependencies; the explicit SDK target fails if its pinned dependency is absent and has no conditional skip. They do not qualify hosted rollout or repair an old unmarked checkpoint. The first full local typecheck found missing checkout SDK links from an install with scripts disabled; the repository link helper repaired that setup. Current-head Linux CI typecheck and build pass. Independent final review checked 12 source scopes and six raw-proof controls with no remaining finding. Fresh Greptile review is 5/5 with no unresolved thread. The full Runner CI suite found two older control-peer test doubles missing the new proof method. Both now return no proof, with all 27 permission, ordering and cancellation assertions retained and passing. The new Cargo-dependent native retry suite uses the existing Rust-cached CI lane. All 39 lane coverage controls pass and prove complete, duplicate-free suite coverage. The first full test run stopped at an admitted chat Retry polling failure. CI also found a stale retained-allocation fixture, now repaired on master by #18984, and a canceled runner job. A real SDK suite initially relied on a dependency absent from the root workspace; the explicit standalone target now owns that acceptance. The separate Product E2E support run had 12 failures in four untouched files. All 12 reproduce on the original clean base with independent dependencies, compiled base source and verified workspace resolution. These concern workflow expectations, cleanup timeout arguments, a profile snapshot and operating receipts. They are not passing acceptance. The later full local repository test command stopped at a 15-second timeout in the unchanged GitHub pre-allocation retry `controller` case. That exact case passed in isolation with unchanged assertions. The broad Runner run initially omitted `test:typescript:prep` and lacked debug fake-provider binaries. Required preparation then passed 370 + 44 package contract cases (11 skipped); the prepared full package run reported 4,932 passed, 9 failed and 13 skipped. All failed families then passed with unchanged assertions in isolation: 8 ACPX controller tests and 2 credential controls. The paired original-base ACPX control also passed. The full-run failures remain recorded; they are not passing full-suite evidence. The first Rust-cached native CI run found a missing generated OpenCode proxy in a fresh checkout. The fixture now runs the existing verified provider-entrypoint builder. Removing the generated proxy reproduces the exact ENOENT before the fix; both real retry cases pass after the fix from the same absent-artifact state. All current-head CI gates pass, including the aggregate verification check. Linux native CI passes all 10 suites: 177 tests pass and 14 opt-in cases skip. Both real provider retry cases pass. The full Runner package passes 297 suites and 4,950 tests, with 1 suite and 10 tests skipped. The CLI workspace groups pass 4,762 and 286 tests. The first CI attempt had Runner shutdowns in the Runner package, Canary and three general server jobs. One unchanged CLI shard also hit child-server startup and dense ZIP deadlines. Both CLI files passed unchanged in isolation: 18 tests. One bounded rerun of unsuccessful jobs and their dependents passed. The original failures remain recorded. ## Risks - Older controllers reject newly written v2 state on downgrade. Current controllers retain v1 restore support, but old or unmarked state cannot gain this new proof. - An older plugin that omits structured ingress evidence remains held. The server and provider must use the new producer contract for this recovery path. The Docker `cloud` target builds the SDK, server/Runner and standalone Daytona producer from one source tree. The default `production` target does not build the standalone producer. Hosted rollout must confirm the new image and active image-owned plugin worker; a registry version alone does not prove its source. An npm-installed producer needs an explicit matching artifact upgrade. - This prevents future proved startup failures. It does not release historical accounting based on an old checkpoint, change retry budgets, replace a model or weaken sandbox isolation. - Zero model usage is separate from process termination, saved files and resource release. Those owners keep their existing gates. ## Model Used OpenAI Codex with tool use and code execution. This runtime does not expose the exact backend model ID or context window to the agent. Read-only independent agents reviewed checkpoint compatibility and SDK/RPC error boundaries. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (affected suites; full local run limits are recorded above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
aaea05e158 |
fix(native): preserve backups when completed warm sessions retire (#18986)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native Runner keeps provider history when a warm session ends. > - Object storage must preserve that history before the sandbox is released. > - A completed turn releases its live controller lease. > - The backup owner lookup rejects that released lease and stops the next task preparation. > - This pull request uses the existing completed-run checkpoint fence for backup storage. > - The next turn can prepare after the previous provider state is saved. ## Linked Issues or Issue Description Refs: #18932, #18011, #18984. **What happened?** A warm Codex turn completed and kept its provider session alive. The next turn stopped during preparation with `native_harness_storage_owner_missing`. The backup lookup required a live lease after normal completion had released it. **Expected behavior** Save the completed session under its retained controller generation and committed result. Keep the existing owner and successor checks. Do not start another provider turn during the save. **Steps to reproduce** 1. Run a warm native session with object-backed provider backups. 2. Commit the first turn and release its controller lease. 3. Start the next task preparation and retire the previous session. 4. Observe the backup owner error before provider work starts. **Paperclip version or commit** The regression is present at `86814c6550f52e99ede150a883bdc773cbb540e3`. **Deployment mode** A hosted server with a remote native Runner and S3 storage. ## What Changed - Accept a released backup owner only when its exact controller record has a committed result. - Reuse the checkpoint owner policy for same-controller preparation and native successors. - Recheck the controller generation, committed result, and released state after upload. - Preserve live-owner rejection and the rule that a late upload cannot replace a newer conversation backup. - Add real PostgreSQL storage and restore regressions. Document the completed-run backup rule. ## Verification - On 2026-11-04, two new regressions failed on the unchanged source with `native_harness_storage_owner_missing`. These covered an idle completed session and a same-controller preparing successor. - After the fix, 54 selected tests passed: 30 durable backup tests and 24 existing early-retirement checkpoint tests. After the fixture-only upstream sync, all 58 selected storage, checkpoint, process, and affected fixture cases passed again. - The storage cases use real PostgreSQL, encrypted archives, object transfer, and restoration. They cover invalid completion evidence, foreign or expired successors, active predecessors, changed generations, changes during upload, and newer backup publication. - Command: `pnpm exec vitest run server/src/services/native-runtime/native-harness-backup-storage.integration.test.ts server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts -t 'released|backup|early retirement|durably retires an accepted warm Runner'`. - An independent source review found no outstanding code issue. - Full repository typecheck passed with `node scripts/typecheck-low-memory.mjs` before the upstream fixture-only sync. Production source is unchanged across that sync. - An assembled process probe passed with the Runner binary built from this head and a fake Codex protocol peer. It started real processes, committed a result, released the source lease, claimed the next controller, and completed the source stop, drain, and suspend commands. It recorded one provider turn start and kept the task bound to its successor. - Command: `pnpm exec vitest run server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts -t 'durably retires an accepted warm Runner'`. - CI exposed the old allocation-readiness test setup in three cases. The branch now includes the merged #18984 fixture correction. The same three cases passed locally after that sync. - An earlier CI test shard was cancelled after its runner received a shutdown signal. Its log was retained before one rerun. Another old-head shard reported failure with no retrievable job log or annotations; its cause remains unknown. Current-head CI must pass independently. - Local `pnpm build` passed at this head. All 1,904 chat cases passed across the supported local shards; the earlier shards used byte-identical source before the fixture-only sync. The interrupted default command is retained and is not reported as a pass. - The local bulk server run had separate enrollment acknowledgment, environment-run status, monitor-settle, idle-history teardown, and repository-rehome failures. In isolation, idle-history passed all 75 tests and the unchanged repository cases passed with their original limits (4.9s large files, 6.4s refs). The remaining four local failures reproduced on unchanged master `04a24f4e06` under the same environment and limits. Their causes remain unproven. Logs and the baseline comparison were retained; the local full suite is not claimed as passed. - Current-head CI first hit the unchanged repository-rehome large-file case's 15s timeout. The same isolated case passed in 4.9s. After source/call-path comparison, one bounded failed-shard retry passed all 115 executed suites; no assertion or timeout was changed. The original timeout remains recorded. - Local verification uses macOS 26.5.2 arm64, Node 26.4.0, pnpm 9.15.4, PostgreSQL 18.1, and Cargo 1.97.1. The current tested head is `622f69f76ff68c71eacdcbcbc6ab4d4c6eb83bf9`. - The full unfiltered [PR workflow](https://github.com/paperclipai/paperclip/actions/runs/41868587974) passed at `622f69f76ff68c71eacdcbcbc6ab4d4c6eb83bf9`: all 12 general server, 3 chat, 9 serialized, workspace, native Runner, build/typecheck, E2E, and final verify lanes. The file selector was skipped, so this was the complete fallback matrix. - Greptile reports 5/5 on this exact head with no open review threads. Disposable local test processes were cleaned up and the source tree is clean. - The real hosted Codex and S3 workflow has not been rerun because the local verification environment has no remote provider credentials. The original task remains unchanged. ## Risks The main risk is a stale controller publishing provider history after a new owner starts. Both storage transactions keep the company, run, boot, and generation fence. A released owner must keep its committed result. Existing successor checks and the newer-backup publication guard remain in force. The live checkpoint writer still requires a non-null lease. No migration or API change is required. ## Model Used OpenAI Codex, GPT-6, with reasoning, repository tools, and code execution. The exact serving model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (focused checks; separate baseline broad-suite failures are disclosed above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
86814c6550 |
fix(recovery): ready retained sandboxes before preparation (#18934)
## Thinking Path > - Paperclip manages work performed by AI agents. > - Restart recovery must preserve each run's workspace and execution owner. > - A retained remote lease can refer to an archived sandbox. > - The reattachment path returned that lease without making it ready. > - A short project-authentication command then had to start the sandbox. > - This PR resumes and verifies the exact retained allocation before preparation. > - Recovery keeps the recorded workspace and preserves saved work if resume fails. ## Linked Issues or Issue Description **What happened?** A native run resumed after a controller restart. Its saved sandbox had been archived. Reattachment returned its lease without a provider lifecycle resume. The next project-authentication probe tried to start that sandbox with its 15-second command budget. The start failed before the probe could inspect project settings. The heartbeat reported a configuration conflict and stopped work. **What did you expect to happen?** Make the exact retained sandbox ready through the existing bounded lifecycle operation. Verify its allocation before running short preparation commands. Keep the validated lease and recorded working directory. Never acquire a replacement in this path. **Steps to reproduce** Hold the provider's exact-lease resume pending and call remote reattachment for an active ephemeral lease. Before this change, acquisition returns immediately. The added tests hold acquisition until resume completes and reject missing, changed or expired allocation evidence. Provider compatibility cases retain the recorded workspace when resume omits directory metadata or returns the allocation root. **Deployment mode** Native Runner in a sandbox with a retained workspace and managed AI connection. The observed provider was Daytona. Refs #18910, which restores selection of retries using current remote workspace storage. This change repairs the next preparation boundary. Related #17421 and #18559 change project-file inspection; they do not make retained sandbox leases ready. ## What Changed - Resume the exact validated sandbox lease before returning it to preparation. - Require the receipt to confirm the original provider allocation without reporting expiration. - Keep the original database lease and recorded directory. Resume metadata can describe the allocation root. - Pass the existing typed resume-directory input through the runtime facade. - Preserve retained E2B sandboxes when resume timeout setup or workspace inspection fails. Keep new-allocation cleanup and explicit release/destroy behavior. - Preserve plugin-task reattachment and lease identity checks. - Update the Stop-accounting fixture to confirm exact-allocation resume before cleanup, with its budget, ownership and no-paid-work assertions intact. - Document the ordering and add readiness, receipt, failure and ownership regressions. ## Verification - Red: seven initial retained-lease cases failed before the readiness change. Two receipt-compatibility cases and four E2B preservation cases failed before their fixes. - Green: all four runtime, workspace-preparation, heartbeat-recovery and orchestrator suites pass: **442 tests**. - The Stop-accounting CI failure reproduced locally. All **five** integration cases pass with the exact-allocation resume fixture. - All **22 E2B tests**, the standalone plugin typecheck and its build pass. - **13 Daytona safety cases** pass, including recorded directories, allocation sentinel rejection, saved-file reads and no replacement creation. The SDK boundary is simulated; the saved-file case uses real files and a POSIX shell. - These focused checks cover **482 passing cases**. - Full recursive typecheck, final server typecheck and full workspace build pass on macOS arm64 with Node 24.22.2 and PostgreSQL 18.1. - Greptile is **5/5** on head `32f915a475275680bcf2a2aa43ae0c72f4e4b1db`. Both original review threads are resolved. - All current-head CI gates pass. The check rollup has 54 successful checks, three skipped checks and one neutral check. - The separate Paperclip Review check remains queued. Its public response says work is paused and the request is saved. The review task is waiting for saved-file recovery, so no Paperclip assessment of this head is claimed. The branch-required gate has passed. - A read-only deployed trace showed an archived sandbox start fail at the short probe's 15-second deadline. Actual provider reattachment with this revision has not been exercised. Live recovery remains unverified until rollout. ## Risks Recovery now makes one bounded lifecycle call before short preparation commands. A missing resume capability or a missing, changed or expired allocation receipt holds recovery. The existing runtime and provider retain allocation and sentinel verification. The path does not acquire, release, replace or reseed the lease. Warm providers can return their existing allocation. E2B retains the existing sandbox if resume setup fails; explicit lifecycle cleanup still owns its removal. No background polling, credential-policy change, migration, workflow change or lockfile change is added. ## Model Used OpenAI GPT-6 through Codex, with reasoning, source inspection, tool use and code execution. The exact backend snapshot and context window were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
84e9c8f364 |
fix(retry): wait for admitted source cleanup before dispatch (#18898)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Connected-task Retry keeps the original request and its current authority. > - One durable Retry can already own a queued successor. > - The old native environment can still have pending cleanup before that successor starts. > - The task execution lock points to the successor, so the old cleanup can escape the queue claim check. > - Dispatch then refuses the source state and fails the new run before provider work. > - This pull request keeps that same run queued and wakes dispatch when cleanup changes. ## Linked Issues or Issue Description Refs #18845, #18853, #18860, #18877, #18882 and #18932. These repairs cover preservation binding, interrupted preparation, old native ownership, manual native holds or other setup guards. This change covers an already admitted connected-task Retry whose native predecessor still owes cleanup. It uses the shared cleanup predicate so the manual-hold repair remains with its existing owner. Search found no duplicate queued-source cleanup repair. **What happened?** A board Retry saved its exact chat request and admitted one queued successor. The source lease was still pending cleanup when that successor reached dispatch. A retained stop receipt did not settle the full cleanup contract. The queue checked the task's current execution owner, which was already the successor. The native authorization guard then rejected the predecessor state and failed the new run before provider dispatch. **Expected behavior** Keep the same canonical Retry and queued run while the exact source cleanup is pending. After authoritative settlement, submit that run through normal dispatch once. Do not infer safe execution from a stop-and-retain receipt alone. **Steps to reproduce** 1. Retry an authorized connected request from a failed native run. 2. Admit one successor while capacity keeps it queued. 3. Leave or restore the source lease to pending cleanup with a matching retained stop receipt. 4. Release capacity and resume the ordinary queue. 5. The old claim path starts the successor and the native source guard refuses it. **Paperclip version or commit** Reproduced against master at `f77499283270d2e966aabd58493f9d35c6bbdf5f`. **Deployment mode** The regression uses the real HTTP route, chat authorization, queue claims, disposable PostgreSQL and the production work coordinator on macOS arm64. A controlled adapter supplies the final result. No hosted provider replay or deployment is claimed. ## What Changed - Validate the exact canonical Retry, chat action, company, task, agent, source and successor before testing source cleanup. - Keep the successor queued under the existing issue claim lock while the shared cleanup predicate remains pending. - Wake only the scoped queued-dispatch worker after cleanup or exact workspace-owner release commits. Add no timer or background scan. - Verify pending cleanup at intake, later cleanup invalidation, one final execution, idempotency, Stop, changed input, unknown stop receipts, a different live controller and signed sandbox absence followed by owner release. - Document the queued-wait contract. No schema, endpoint, credential, model or provider policy changes are required. - Align five native preparation acceptance fixtures with the existing workspace publication contract. Use the real reservation and publication owner. Retain a raw-root refusal case with no provider launch and preserved files. ## Verification - Latest master sync at `e26cad76b8f20ba823d702ed60e7b7294f8f5a7c` includes master `396b7fc62a0a4b1d8aadbda8221c05879d54177c`. The shared test helper preserves #18944's canonical retained-directory default, the real publisher for the five fresh-workspace cases, and the explicit raw-root refusal. Independent remerge review found no dropped upstream behavior, guard or assertion. - The latest head passes the focused six cases, all 580 complete workspace/continuation tests, all 38 selected HTTP Retry cases, fresh full recursive Node 24.22.2 types and full build. No affected tests skipped. An initial cached test-layer attempt read old budget declarations; regenerating them resolved all seven errors without test/mock or production-export changes. - All current-head CI is green at `e26cad76b8f20ba823d702ed60e7b7294f8f5a7c`, including required [ci / verify](https://github.com/paperclipai/paperclip/actions/runs/41846304017/job/126636751314) and [ci / e2e](https://github.com/paperclipai/paperclip/actions/runs/41846304017/job/126636638086). Greptile and Paperclip Review both rate this exact head 5/5, with no findings or open review threads. - Master sync at `3e4f266c2da172a84afd0b9cb4a91ea3b95f3c0d` merges master `b891e69bd23d6dab8ce536569e15bc4a7a31c35c`. The only conflicts were adjacent queue imports and two HTTP Retry test matrices. Both imports and both complete matrices remain unchanged. Independent review found no relaxed guard or assertion. - Node 24.22.2 and pnpm 9.15.4: all 38 selected real HTTP Retry cases, 194 queue/cleanup/finalizer cases, module boundaries, recursive types and build pass at `3e4f266c2da172a84afd0b9cb4a91ea3b95f3c0d`. - That sync head's [server check](https://github.com/paperclipai/paperclip/actions/runs/41838741852/job/126604742761) failed five native acceptance cases in CI and locally with `remote_workspace_published_path_invalid`. The test fixture supplied a raw directory instead of a fresh publisher-generated workspace. The guard from #18932 remains unchanged. The same five cases passed earlier Linux CI before that guard was present. - The test-only alignment at `dcdee1bb6c2e2e6dd494382499d17c23a3ce71a8` obtains the saved authority, path and run reference from real `reserveRemoteWorkspace` and `prepareRemoteWorkspace`. All five original cases and a raw-root no-launch/no-result/files-preserved control pass. Their native result, file, catalog and single-execution assertions remain intact. An initial setup attempt was refused for a noncanonical allocation root; the fixture now resolves that owned root before publication. - At previous fixture head `dcdee1bb6c2e2e6dd494382499d17c23a3ce71a8`, the generated server test-layer typecheck passed. All 580 tests in the complete workspace/continuation family pass, with no skips. That test-only commit left production files byte-identical to the sync head. - Previous published-head CI was green at `dcdee1bb6c2e2e6dd494382499d17c23a3ce71a8`, including required `ci / verify` and `ci / e2e`. The [Linux Node 24.22.2 server shard](https://github.com/paperclipai/paperclip/actions/runs/41842665765/job/126618334638) executed all five original acceptance cases and the new raw-root refusal case successfully. Its whole workspace recovery file passed all 245 tests. - Greptile reviewed previous published head `dcdee1bb6c2e2e6dd494382499d17c23a3ce71a8` at 5/5, with no findings or open review threads. No unchanged-job rerun was used for the reproduced sync-head failure. - Causal red: the real HTTP regression started and failed the successor at the native source-state guard while cleanup remained pending. A second red proved that signed sandbox absence followed by copyback-owner release lacked a queued-dispatch wake. - Green: seven HTTP/PostgreSQL cleanup cases, the exact confirmed-owner release notification control and seven existing retry lineage controls pass. The real coordinator resumes from committed cleanup or workspace-owner release, without a manual queue restart. One controlled adapter execution produces the result after settlement. - Earlier branch verification passed full repository `pnpm -r typecheck`, `pnpm build`, module boundaries and `git diff --check`. - The full stale-queue file control passed all 64 cases with unchanged observer/hook limits. The initial broader affected run had 192 passes and one non-assignee native-owner case miss its existing five-second adapter observer while already running with no error. The isolated case passed; the unchanged base passed all 64 cases but hit its existing 30-second teardown timeout. Those original results are retained. The new queue hint has no dispatch consumer in that fixture. - Earlier-head CI was green: 54 success, three skipped and two neutral checks at `6adb9f1ba682282c512d5e751ba2aa663f614ca2`. One chat job was cancelled by an external runner shutdown; its single unchanged-job rerun passed. No source was changed to obtain that result. - Greptile and Paperclip Review both rated earlier head `6adb9f1ba682282c512d5e751ba2aa663f614ca2` 5/5 with no findings or unresolved threads. These results do not qualify the master-sync head. - At initial verification, a complete local `pnpm test:run` was not repeated while the related setup repair's full local run was active. The full earlier-head CI suites passed. A full local suite has not been repeated for the mechanical master sync. Local verification uses the affected HTTP, producer, lineage and queue suites. No full local-suite pass is claimed. - The fresh clone's missing plugin SDK links were repaired with the repository's own postinstall helper before verification. The embedded PostgreSQL package's own symlink helper was also required. - No staging task was submitted again with this repair, and no deployment is claimed. ## Risks - A pending source cleanup can keep the queued run waiting. Its cleanup owner still owns reconciliation and its existing limits. - Stop, changed source, unknown termination and live ownership retain their existing authority checks. A cleanup notification grants no execution authority. - A new cleanup change after claim still faces the existing final dispatch guard. - Cleanup notifications can change dispatch timing in a running controller. The real coordinator cases verify commit-bound wakeups; the broader current queue-file control passed all 64 cases. The earlier loaded-run and baseline teardown failures are retained in verification. ## Model Used OpenAI GPT-6 through Codex, with reasoning, code editing, tool use, PostgreSQL and browser verification. This session does not expose a more specific deployment model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
5d226ca08a |
perf(chat): reuse bound-run authorization query templates (#18966)
This PR depends on #18969. Merge that fixture-only prerequisite first, then retarget this PR to master. No merge or deployment is requested in this work. Its base is `codex/fix-recovery-published-workspace-fixture`; the six-line recovery fixture repair is excluded from this diff. ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - External chat authorization binds each run to current messages and conversation policy. > - Review wait and attention-retention paths repeat the same four-table read. > - Each call rebuilt the Drizzle query even when its SQL shape stayed the same. > - This pull request reuses the compiled template for the exact database handle and lock mode. > - Fresh parameters and current database reads preserve authorization while reducing Node CPU work. ## Linked Issues or Issue Description **What existing behavior does this improve?** The `authorizeChatConversationForBoundRun` links read used by external chat waits and attachment reuse. **Current behavior** At baseline `e54f9fbf828e06e385e0c62dc381ccb46190aa30`, each call rebuilt and serialized a four-table query. The observed staging shape accounted for 16.2% of sampled SQL serialization CPU across 32 samples. This is a share of query compilation CPU, not total server CPU. **Proposed behavior** Compile each read, blocking-lock or NOWAIT variant once per exact live database or transaction object. Bind new company, issue, agent, provider and comment IDs on every execution. Continue all current database reads and authorization checks. **Reason and benefit** Repeated authorized service calls used 26.91% less Node process CPU in the matched local fixture. All seven CPU pairs improved. This reduces repeated compilation on the shared service path. Related: #18896 retains task Retry intent through identity contention. It changes a different workflow and does not reuse this query template. A public issue and PR search found no duplicate template change. ROADMAP.md has no conflicting planned feature. ## What Changed - Keep compiled queries in a `WeakMap` keyed by the exact supplied database or transaction object. Create only the requested lock variant. Never cache a result or authority decision. - Bind the comment list as `ANY(uuid[])`. This supports changing list sizes with only three templates per live handle. Keep all joins, selected fields, policy checks and lock clauses. - Add real PostgreSQL regression tests, a reusable fixture, an actual service benchmark and a dated performance note. ## Verification - Passed 17 real PostgreSQL tests: `node node_modules/vitest/vitest.mjs run --project @paperclipai/server server/src/__tests__/prepared-chat-authorization.integration.test.ts server/src/services/native-runtime/chat-attachment-reuse.test.ts`. - After rebasing, seven selected existing external-chat-wait integration cases passed (266 deselected). They cover small current review waits, answered-chat identity advisory locks, connection-disable serialization and contended endpoint proof retry. - New tests cover changed parameters and list sizes, duplicate/empty inputs, invalid UUID errors, company/agent/provider binding, current setup generation and credential fingerprints, root/transaction/savepoint isolation, rollback, read mode during a held row lock, blocking timeout and NOWAIT contention. - Actual service benchmark: `node --import ./server/node_modules/tsx/dist/loader.mjs server/scripts/benchmark-bound-chat-authorization.ts --baseline-ref=e54f9fbf828e06e385e0c62dc381ccb46190aa30`. Node 24.22.2, macOS arm64, real PostgreSQL, 500 warmup calls per implementation, seven alternating pairs of 500 calls. Every result matched. - Pinned measured treatment `8386eb78d118faaf6c2fd7f7471b38c0f4b2ef54`. Its source is unchanged after rebasing onto master. Median Node CPU per 500 calls: 379.033 ms to 277.048 ms, down 26.91%. Every CPU pair improved by 13.54–33.52%. Median wall time: 742.109 ms to 648.005 ms. One wall pair increased by 0.51%. - A separate diagnostic after timing found five SQL compilations per baseline call and four per warm treatment call. All five database reads still run. - The final four-test PostgreSQL authorization suite passed again after rebasing and strengthening the lock proof. It holds only a message-link row, so principal advisory contention cannot mask a missing joined-query lock. - Independent source review found no correctness issue in exact-handle ownership, fresh binding or the three lock variants. - Preserve setup failures: the first test attempt lacked plugin SDK build output. Shared and SDK builds fixed it. Later attempts could not start embedded PostgreSQL because its local dependency directory lacked bundled native-library aliases. `DYLD_LIBRARY_PATH` did not fix it. Restoring package-local aliases fixed the environment; the final 17 tests then passed. No dependency or lockfile change is included. - The first full typecheck found new test-only errors: PostgreSQL transaction handles lack the root database `$client` type, and runtime setup test data includes fields outside the saved setup interface. Commit `40a4645750` uses the existing explicit transaction test cast and SQL JSON fixtures. No production contract changed. The full check also found local plugin SDK link prerequisites, which the coordinator is repairing separately. - Preserve the first Linux CI failure: [server shard 9](https://github.com/paperclipai/paperclip/actions/runs/41837775564/job/126602280304) failed the new setup-fence fixture's allowed case. The fixture supplied a client `processedAt` while PostgreSQL supplied default `receivedAt`; this could violate the required processing order. The test now uses explicit ordered timestamps and no wall-clock window. The four-test PostgreSQL suite passed again in 19.93 seconds. Production guards were not weakened. - [Server shard 2](https://github.com/paperclipai/paperclip/actions/runs/41837775564/job/126602280291) also failed five workspace-recovery cases with `remote_workspace_published_path_invalid`. Those failures reproduced on both pure master and the exact CI merge with the chat optimization removed. #18969 fixes the canonical path in that separate fixture, with five failing cases before and five passing after. This PR is stacked on that independent repair for final CI. - The additional Paperclip review reported a node-postgres statement-name collision and requested explicit pool cleanup. This repository uses postgres-js; locked Drizzle 0.45.2 accepts the Drizzle label but does not use or forward it. [The review response](https://github.com/paperclipai/paperclip/pull/18966#issuecomment-6632666599) links primary driver source and the database factory. Commit `c5d0d84f36` adds explicit pool teardown in `try/finally` and a real same-transaction `read → blocking → nonblocking → read` case. All five authorization tests passed in 6.47 seconds. The original separate row-lock test remains. Production SQL and code did not change. - Final combined local validation at `4ab58ff70b` passed full `pnpm -r typecheck` (39.864 seconds) and 35 real PostgreSQL cases across the chat, GitHub and shared prepared-query suites (11.378 seconds). Full `pnpm build` passed earlier with byte-identical production source; later changes affect tests only. The production file SHA matches the benchmark and this PR. The original full `pnpm test:run` was interrupted after all six chat shards passed and general tests had partly run, so it is not a full local pass. Final current-head CI provides the broader coverage. A sandbox retry failed with EPERM before assertions; the identical escalated commands then passed. Final head passed CI in [workflow 41843806060](https://github.com/paperclipai/paperclip/actions/runs/41843806060), including all build, typecheck, general, serialized and e2e test gates. The previous head `40a464575009dfc3cce7afa1eb4fffa5540dcaf6` passed Greptile 5/5, security review, typecheck, build and canary validation. The prior stacked head `74cb36370162750f8e135ed091603d3629d23b0d` passed full CI, Greptile 5/5, policy and security checks. Final head `c5d0d84f369d897a66d9b6441bab10cb79f593b9` passed a fresh Greptile 5/5 review with no actionable findings, security and PR policy checks. Its complete [current-head CI workflow passed](https://github.com/paperclipai/paperclip/actions/runs/41843806060). All 54 required CI, Greptile, policy and security checks passed. The optional Paperclip Review check is queued for this head; no current-head score is claimed. Production source SHA-256 remains `72ce9d2317e5c97b36e299e24e30fc1aa63020d8c266e495fc45dbfdd5331e93`, identical to the measured code. ## Risks - `ANY(uuid[])` relies on postgres-js array binding and UUID casting. Real PostgreSQL tests cover single and multiple valid values and invalid UUID errors. - A new transaction or savepoint pays its first compilation cost. The weak cache cannot reuse a root session for it. This benchmark covers repeated calls on one root handle. - The team paused other work during the benchmark. Unrelated Rust compilation was active elsewhere on the host. Process CPU excludes those processes, but host contention can affect caches, scheduling and wall time. This source-mode local result does not prove production request latency or whole-server CPU savings. - No migration or API contract changes. No deployment or merge is included. Monitor authorization failures and lock-contention behavior after an authorized rollout; revert the narrow service change if those regress. ## Model Used OpenAI `gpt-6.1-sol` through Codex, with reasoning, repository editing, tool use and test execution. The host did not expose a context-window size or deployment snapshot ID. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |