mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-09 07:10:34 +01:00
2f35eb10806db4d35d0e366b4e414df74bb9c097
532
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Devin FoleyandPaperclip
|
11bc490cda |
fix(recovery): keep a stopped copy-back's files instead of holding the workspace (#19385)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native run can have its result accepted before the copy-back of its sandbox files finishes. If the copy-back fails, it retries. > - A board Stop during those retries ends the run. It sets the run's finalization to `terminal_failure` / `native_retry_cancelled`, and the result stays accepted. > - After that, nothing exports the files. The finalization worker skips terminal rows, and cancelled-workspace preservation skipped any run with an accepted result. > - The task workspace stays dirty and bound to the stopped run. Every later run on the task waits for the workspace, then fails setup with the unfinished-writer error. > - This pull request lets cancelled-workspace preservation also admit an accepted result whose copy-back a Stop ended. > - The benefit is that stopping a stuck copy-back saves its files and frees the task, like stopping any other run. ## Linked Issues or Issue Description Refs #16700, #17677, #18820, #19304 and #19305 (cancelled-workspace preservation). This pull request adds the accepted-result case to that path and keeps its other checks unchanged. **What happened?** On a hosted instance, a native run's result was accepted, but its copy-back kept failing and retrying. A board operator pressed Stop to unblock the task. The run became `cancelled`, its sandbox was stopped and kept, and its finalization moved to `terminal_failure` / `native_retry_cancelled` with the result still accepted. The next run on the task was cancelled with "Shared execution workspace is busy: run … ended, but its workspace save has not settled". About 24 automatic retries followed over an hour. The last one failed with "The workspace still has an unfinished writer or save operation." After that nothing ran: the task stayed `in_progress` with no run and no recovery action. A Retry or a new comment would hit the same block. The workspace stayed `dirty: true` with the stopped run as its writer, and no `workspaceRestoreRecovery` was ever recorded. The reason: `preserveCancelledNativeWorkspaces` requires `resultId IS NULL`, and `registerCancelledWorkspacePreservationWork` and `preserveNativeFailedWorkspace` both return early when `resultId` is set. **Expected behavior** Stopping a run during its copy-back retries keeps its files, as stopping any other run does. Paperclip's workspace recovery saves them and releases the workspace. Nothing commits or replays the stopped result. **Steps to reproduce** 1. Accept a native result for a run whose copy-back is in `retryable_failure`. 2. Stop the run with the board Stop. The finalization becomes `terminal_failure` / `native_retry_cancelled`. 3. Start another run on the task. It waits for the workspace and then fails setup. No save of the stopped run's files is ever recorded. The new database tests run steps 1 and 2 against PostgreSQL and check that the save is recorded. **Paperclip version or commit** `master` at `9ceaec6f97e`. **Deployment mode** Any deployment that uses remote (sandbox) task workspaces. Seen on a cloud-managed instance. ## What Changed - `server/src/services/native-runtime/native-failed-workspace-recovery.ts`: cancelled-workspace preservation now also admits an accepted result when the finalization is `terminal_failure` with `native_retry_cancelled`. That is the state a Stop leaves after ending copy-back retries. All three entry points use the same rule: - the startup scan `preserveCancelledNativeWorkspaces`; - the worker `registerCancelledWorkspacePreservationWork`; - `preserveNativeFailedWorkspace`, only when it is called with `requireCancelledStop`. Preservation still requires everything it required before: a cancelled run, an acknowledged Stop, no live finalization owner or controller lease, a prepared workspace reference, and the exact retained source. - An accepted result with any other failure code is still skipped; those runs keep their own export repair path. Failed (not cancelled) runs are unchanged. - The accepted result stays recorded and uncommitted. Preservation does not change the task's status or queue a run. - `doc/execution-semantics.md`: describe the stopped copy-back case in the paragraph on writers stopped mid-run. - `native-cancelled-workspace-preservation.test.ts`: move the accepted-result fixture into an `acceptResult` helper, and add a test with three cases. ## Verification - Failing first. The new test "preserves files of an accepted result whose copy-back was stopped" has three cases: worker, startup scan, and a sandbox that the Stop already stopped and kept, through the real `prepareSandboxStopAndRetain` and `settleStopOnlyCleanup`. All three fail on `master`, with no `workspaceRestoreFailure` recorded, and pass with this change. - Each case checks that: - the save obligation and recovery action are recorded for the exact lease; - `inspectWorkspaceRecoverySource` finds one source; - the finalization keeps its `resultId`, `terminal_failure` and `native_retry_cancelled`; - the workspace stays held by the stopped writer until the save publishes; - no run or wake is created, and the task status is unchanged. - The existing "does not admit preservation for accepted-result" case still passes. It uses an accepted result with a different failure code. - `cd server && npx vitest run src/services/native-runtime/native-cancelled-workspace-preservation.test.ts`: 44 passed. - Related suites, 10 files and 854 passed: - `native-failed-workspace-recovery`, `workspace-recovery`, `heartbeat/run-control` - `native-cancellation-request` (unit and integration) - `heartbeat-remote-workspace-busy`, `native-workspace-export-retry`, `native-workspace-reconstruction` - `native-stop-accounting`, `task-workspace-binding` - `cd server && npx tsc --noEmit -p tsconfig.json`: pass. - Not run end to end on a live instance. Diagnosed from that instance's run, finalization, lease, workspace and recovery-action rows. ## Risks - On deploy, the startup scan and the worker will pick up existing cancelled runs in this state and record their saves. That is the intended repair, and it uses the same retention checks as other stopped writers. - A copy-back interrupted by a server restart can leave a `remote_workspace_snapshot` operation marked `running` after its process is gone. This change does not settle such rows. If one blocks the later save, the save follows its existing retry and board hand-off. - No schema migration. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) through Claude Code, with tool use (shell, file edits, test runs, read-only database queries). The runtime does not expose the context window or reasoning configuration. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
6ca7b151f7 |
feat: compact an agent's context from the task Properties panel (#19378)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A task's Properties panel shows the assignee agent's context window
(paperclipai/paperclip#19273)
> - The board can see that the context is nearly full, but it cannot act
on it
> - A full context makes the agent compact at an unplanned time, often
in the middle of work
> - This pull request adds "Compact now" to the Context hover card
> - The benefit is that the board can compact the agent's context
between runs, at a time it chooses
## Linked Issues or Issue Description
Refs paperclipai/paperclip#19273
**Problem**
The Context row shows how full the agent's context window is. The board
cannot ask the agent to compact it.
**Proposed solution**
A "Compact now" button in the Context hover card. It runs the harness's
own compaction: Codex `thread/compact/start`, and Claude's `/compact`
command through ACP.
**Alternatives considered**
A separate PRP `session.compact` command. Both harnesses run compaction
as a turn, so a compaction turn keeps the existing durable turn binding,
interruption, and recovery. A separate command needs a second turn
lifecycle.
## What Changed
- UI: the Context hover card shows Compact now with three states:
available, Compacting…, and unavailable with a reason (agent mid-run,
harness unsupported, task closed). The Storybook story shows each state.
- Server: `POST /api/issues/:id/context/compact` (board only,
`decideIssueAccess` `issue:mutate`, activity log
`issue.context_compaction_requested`). It queues a short run marked by
`contextSnapshot.contextCompactionRequestId`. `GET
/issues/:id/context-usage` returns the `compaction` state.
- Server: a compaction run does no task work. The native status arbiter
preserves the task status. The lifecycle skips handoff, liveness
continuation, review-path disposition and the issue comment, as it does
for goal-control runs.
- Runner TS: `startTurn({ compact: true })`. The Codex harness session
sends `thread/compact/start`. The runnerd transport maps it to PRP
`turn.start` with `compact: true` (Codex) or `text: "/compact"`
(Claude/ACPX). The native runtime option `sessionCompaction` requires a
resumed session and settles with a synthesized yielded result.
- Runner Rust: `turn.start` accepts `compact: true` for Codex.
`CodexProvider::start_compaction_turn` sends `thread/compact/start` and
binds the turn from the `turn/started` that follows the empty response.
- Fix (from #19273): Codex emitted `context.compacted` on both the start
and the end of one compaction item. It now emits it once, and the count
is distinct by `compactionId`.
- Fix (from #19273): the conversation event view dropped
`context.compacted`, so a finished run never showed the divider. The
view now keeps compactions, plus context readings for runs that
compacted. A finished run's compactions show as thread dividers.
- Fix: the Codex harness stamps before/after sizes on the compaction
event. Codex reports the compacted size before the compaction item ends,
so the old neighbour-based guess showed the wrong size.
## Verification
- `pnpm -r typecheck`, `pnpm build`, `pnpm check:token-gates`, `node
scripts/storybook-catalog.mjs`: pass.
- New tests: `server/src/__tests__/context-compaction.test.ts`,
`status-arbiter.test.ts`, `issue-context-usage.test.ts` (duplicate
compaction counts once), `ui/.../context-usage-row.test.tsx`,
`native-session-runtime.test.ts` (context compaction), Codex driver and
runnerd transport tests, Rust
`codex_compaction_turn_binds_the_turn_reported_after_its_empty_response`.
Rust `codex_provider` suite: 115 passed.
- `pnpm test:run` locally: one failure in
`chat-channels.integration.test.ts`. It reads
`.claude/skills/slack/SKILL.md` from the main checkout, which is not
present in this worktree. It is not related to this change.
- Live, local worktree instance, Codex agent on the Paperclip runner:
clicked Compact now. The run succeeded. Context went from 35k to 9.4k
tokens, and later 9.4k to 9.2k. The task stayed In Review. The thread
shows "Context compacted · 9.4k → 9.2k tokens" and the meter dropped.
- Live Claude proof: pending. The local Claude agent needs an Anthropic
API key. claude-agent-acp handles `/compact` as an echo-less turn, and
the runner already maps its "Compacting completed." notice to
`context.compacted`.
## Risks
- A compaction run resumes the agent's session. If the session cannot
resume, the run fails with `native_context_compaction_unavailable` and
the task is unchanged.
- Compact now is not available while the agent runs on the task, because
compaction interrupts the active turn.
- The conversation view now returns context readings for runs that
compacted. Runs that never compacted return the same events as before.
- An old runner binary ignores `compact` and rejects the `turn.start`
because it has no text. It does not start an ordinary turn.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code
with tool use and code execution.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandClaude Opus 5.5
|
240d46e2ec |
fix(native-runtime): join the deferred observer seen when a duplicate append arrives (#19397)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - The native runtime writes runner events through
`PaperclipControlPlanePort.appendEvent`.
> - A terminal event can defer its post-commit observer. A duplicate
append of the same event must join that observer. It must not run
`onDuplicateEvent`.
> - The duplicate looked up the observer only after its own event
transaction finished. If the original observer settled during that
transaction, the observer left the map. Then the duplicate ran
`onDuplicateEvent`.
> - This made the test "joins the exact deferred observer on duplicate
append..." fail on slow CI (job 127633060467, and once on Windows).
> - This pull request captures the pending observer when the duplicate
arrives. The join decision no longer depends on database latency.
> - The benefit is a deterministic join and a stable test.
## Linked Issues or Issue Description
**What happened?**
The server test `paperclip-control-plane-port.test.ts > joins the exact
deferred observer on duplicate append and replays it after a new port
opens` failed intermittently with `Error: must join the original owned
observation`. The test gives the duplicate append 30 ms and then
releases the original observer. On a slow runner, the duplicate's event
transaction took longer than 30 ms. The original observer settled and
its `.then` handler removed the map entry. The duplicate then found no
observer after its transaction and ran `onDuplicateEvent`. (The CI
failure took 90 ms.)
**Expected behavior**
A duplicate append that arrives while the original deferred observer is
pending joins that observer. Database latency does not change this
result.
**Steps to reproduce**
1. Add a 60 ms sleep inside the `appendEvent` transaction when the
receipt disposition is `duplicate`.
2. Run `pnpm vitest run
src/services/native-runtime/paperclip-control-plane-port.test.ts -t
"joins the exact deferred"` from `server/`.
3. The test fails with `must join the original owned observation`. This
is the same error and stack as CI.
**Paperclip version or commit**
`master` at
|
||
|
DottaandClaude Opus 5.5
|
958f9aa9f3 |
feat(issues): limit agent-created task trees to parent and child (#19312)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agents split work into tasks. A task can have a parent task. > - Today agents can nest tasks to any depth: parent, child, grandchild, and deeper. > - Deep agent-made trees are hard to follow, and an agent that works on a child task makes grandchildren by default. > - This pull request limits agent-created trees to two levels: a top-level task and its children. > - A request for a grandchild does not fail. The server puts the new task beside the requester and tells the agent. > - Board users keep full control. They can still nest to any depth. Existing deep trees stay valid. > - The pull request also records which task created each new task, so the audit trail stays complete when the parent changes. ## Linked Issues or Issue Description No public issue exists. This is the feature description. Related: #16853 (budgets that include descendant work) also reasons about task trees; this change does not alter its traversal. **Subsystem affected** Issues (tasks): create, child create and move in the issue service and routes. Agent tools, chat suggestions and creation provenance use these paths. **Problem or motivation** Agents create subtasks as deep as they want. An agent that runs on a child task gets that child as the default parent, so its follow-ups become grandchildren. The trees become deep and hard to read. The parent link is also the only record of which task created a follow-up, so that record is lost when the parent changes. **Proposed solution** Limit agent-created trees to a parent and its children. When an agent asks for a grandchild, put the new task under the requester's parent and return `parentAdjusted`. Do not reject the request. Keep board users unrestricted. Record the originating task in a new server-set `origin_issue_id` column. Keep `origin_run_id` as the originating run. **Alternatives considered** Reject grandchild requests with an error. We did not do this, because agents then fail mid-task. Add a database constraint. We did not do this, because existing deep trees must stay valid and nesting can return later. ## What Changed - Add `MAX_AGENT_TASK_HIERARCHY_DEPTH` (2) and the `IssueParentAdjustment` type in `packages/shared`. - Add `resolveNewTaskParent` and `planIssueParentChange` in `server/src/services/issues.ts`. `create`, `createChild` and `update` use the agent limit by default (`parentDepthPolicy: "flatten"`). - The issue create, `/children` and PATCH routes pass `allow_nested` for board users. For agents, the routes check authorization against the parent that the server actually writes. - The requester keeps its behavior when the server moves a task beside it: `blockParentUntilDone` blocks the requester, and the new task inherits the requester's workspace and privacy. - The server never moves a task above a low-trust boundary root that contains the requester. - These subsystems keep any depth, because they find their work by parent: watchdog reviews, monitor recovery, email-channel placement, and routines that a person configured. - An agent move of a task that has children skips only the parent change and returns `reason: "has_children"`. - Add `issues.origin_issue_id` (nullable, `on delete set null`, indexed) in migration `0434`. Only the server sets it. The `createdFrom` detail field, the `createdFromIssueId` filter and task-creation provenance read it first. - Chat prompts now let a chat agent create one task, a task with children, or separate tasks. The conversation is never a parent. - The runner `create_task` tool accepts an optional `parentTaskId`. A chat agent can create a top-level task and then put subtasks under it. The run must be able to read and change the parent. A conversation is never a parent. The depth limit still applies. Project tools map the field to `parentId` through the issue route. The protocol manifest, semantic contracts and eval fixture are regenerated. - Watchdog runs that have an active watchdog keep their requested parent on all create paths, because their access is limited to the watched subtree. - Fix a bug that the test drive found: the chat (MCP) `create_task` ignored the requested `status`, so a `backlog` task became `todo` and started to run. It now uses the same rule as the native tool. - Update `skills/paperclip`, `doc/SPEC-implementation.md`, and add the plan `doc/plans/2026-11-05-two-level-task-hierarchy.md`. ## Verification - `cd server && npx vitest run src/__tests__/agent-task-hierarchy-depth.test.ts src/__tests__/agent-task-hierarchy-depth-routes.test.ts`: 19 new tests for service and route behavior, all pass. - Related suites (64 files: issues service, thread interactions, privacy, watchdogs, provenance, runner tool authority, decisions, routines, project tools): 1208 tests pass. Two tests changed because of the intended behavior and pass now. Two `email-channels.integration` tests fail locally because the test cannot find a bundled skill file from this worktree path. This change does not touch that code. - `packages/db`: migration safety, snapshot drift and journal tests pass. - Typecheck passes for `server`, `packages/shared`, `packages/db`, `cli` and `ui` (local `ui` errors come only from generated Storybook catalog files that are absent). - Local test drive on an isolated instance with real Claude agents: 29 of 29 use cases pass, including an agent on a child task (follow-ups placed beside it, with the notice) and a chat that creates a top-level task with three subtasks. - I did not run the full `pnpm test:run` or `pnpm build` locally. CI runs them. ## Risks - Behavior change for agents: some follow-ups that were grandchildren are now siblings. The parent of the requester is woken when all its children finish, which now includes these tasks. - The runner `create_task` input contract adds one optional field, so the protocol manifest and the pack contract change. The tool output does not change: it shows the actual `parentId`, and the activity log records `parentAdjusted`. - Migration `0434` adds a nullable column, a foreign key and an index. It does not backfill rows. - `IssueCreationSource.run` is now nullable. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, local test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
36643d9378 |
fix(native): retire finished owners whose continued turn rotated authority (#19373)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - The native runner runs per-turn agent sessions in managed sandboxes.
A controller keeps a durable session root for each session.
> - When a per-turn owner finishes without a saved runner checkpoint,
the next turn retires that owner. Retirement first checks that the
session root holds only known entries.
> - The runner transport writes two entries that the check does not
know: `authority-epochs/` and `runtime-context.json`. A continued turn
writes `authority-epochs/` when it rotates authority from the turn
before it.
> - So the third turn of an ephemeral-lease task cannot retire the
second turn's owner. The run fails with
`native_continuation_requires_reconciliation` and the task waits for a
user message.
> - This pull request classifies every session-root entry in one
contract module, and adds a test that scans the writers so the list
cannot drift again.
> - The benefit is that follow-up turns continue without manual help,
and any future refusal names the unexpected entry.
## Linked Issues or Issue Description
Refs #18846, Refs #19061, Refs #19355
**What happened?**
On staging, the third turn of every task for one agent (gpt-5.6-sol,
native runner, ephemeral sandbox lease) failed. The error was
`runner_state_identity_mismatch:
prior_owner_terminal_state_indeterminate;
retirement_refused=unexpected_root_entry`. The first two turns
succeeded. The task then went to `blocked` with "Saved work recovered.
Send a message to continue."
**Expected behavior**
The third turn retires the finished second-turn owner and continues the
conversation.
**Steps to reproduce**
1. Assign a task to a native agent that uses an ephemeral sandbox lease.
2. Let the first turn finish. Reopen the task with a comment so a second
turn runs and finishes.
3. Reopen the task again. The third turn fails with the error above.
**Paperclip version or commit**
Staging near 9a79384958; reproduced locally on master
|
||
|
DottaandPaperclip
|
7c0922a7b6 |
fix(native): preserve provider backups during slow workspace saves (#19359)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Remote native sessions save provider state so another web server can continue the same conversation. > - A provider can suspend while its accepted result is still saving a large workspace. > - The run has released its live lease at that point. A new backup-owner lookup rejects this uncommitted state. > - This pull request captures the real owner during backend creation and retains it with the current session authority. > - The existing database checks still reject a changed boot, generation, result, or successor before inspection and during publication. > - The benefit is that a slow workspace save no longer prevents the provider backup. ## Linked Issues or Issue Description **What happened?** A remote session can fail to save its provider backup with `native_harness_storage_owner_missing`. The provider has suspended, but its accepted result is still in `workspace_finalizing`. The close callback tries to acquire a new owner after the run released its lease. A later allocation cannot use a backup that was never published. **Expected behavior** Save the suspended provider with the authority acquired during execution. Recheck that authority before provider inspection and before and after upload. Keep result finalization and provider-work permissions unchanged. **Steps to reproduce** 1. Start a remote native session with durable provider storage. 2. Accept its result and release the live lease while its workspace save remains pending. 3. Close the provider transport. The old owner lookup rejects the save before provider inspection. **Paperclip version or commit** Reproduced on master `bf10bc695abaef6b20e85b287f456b5ce2edb266`. **Deployment mode** A web server with object storage and remote managed Runner sessions. Related: #19225 retains failed-turn ownership through close. #18436 restores an external controller checkpoint before a changed warm process is replaced. This change fixes late backup-owner acquisition while an accepted result saves its workspace. ## What Changed - Capture the provider backup owner when the remote backend still owns its execution lease. - Bind that owner to the current session authority. An older close callback selects the current warm attachment's owner. Revalidate the captured fence before inspecting or skipping provider state. - Add a regression for the close callback during workspace finalization. - Restore encrypted provider files with a captured owner while workspace finalization remains pending. Reject boot, generation, result, and successor changes during upload. - Document the ownership contract. ## Verification - Observed the callback regression fail on the old code with `native_harness_storage_owner_missing`. The same test passes with this fix. - `cd server && node ../node_modules/vitest/vitest.mjs run --config vitest.config.ts src/services/native-runtime/native-harness-backup-storage.integration.test.ts`: 59 passed, using real PostgreSQL and encrypted object write/readback. - `node scripts/check-module-boundaries.mjs`: passed. - Full workspace typecheck: passed with the complete production fix. - Native execution: 1,151 passed. Managed warm lifecycle: 9 passed. These tests require a lost-owner error during shutdown, including before an incomplete provider checkpoint is skipped. The final change to the lifecycle test also requires rejection after lease renewal is lost. - Full local `pnpm build`: passed on the final head. - The full local `pnpm test:run` stopped in chat shard 6 on five existing deadline assertions. Earlier chat shards passed; the PR CI chat shards passed. This is not a full local-suite pass. - The first CI server shard 4 attempt failed the unchanged legacy environment-inheritance fixture at its five-second completion wait. The same failure reproduced locally on the repair head and on an earlier checkout without this repair. One unchanged CI retry passed. No fixture assertion or timeout change is shipped. - Latest-head CI has passed. Greptile reviewed the final head at 5/5 with no open threads. The separate Paperclip Review is paused. This PR does not claim staging acceptance. ## Risks The backup owner now lives with its session authority. A wrong authority selection must fail rather than publish provider state. Database publication still checks the original company, task, run, boot, generation, and accepted result before and after upload. Tests change the owner during upload and verify that the selected backup stays intact. No schema, data migration, or new provider-work permission changes. ## Model Used OpenAI Codex, GPT-6 model family, with reasoning and code execution tools. The host does not expose the exact runtime model ID or context-window limit. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Nicky LeachandClaude Opus 5.5
|
a6ab6a301c |
fix(server): retry a run when its remote Runner is out of date (#19326)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - An agent can run on a remote Runner that an environment plugin
provides
> - A release can update the app before the remote Runner, so for a
short time the app sees an older Runner
> - In that window the plugin can refuse the task, but Core cannot tell
this refusal from other errors
> - Today the run fails as `setup_failed` with no retry, and the task
goes to board recovery. Before #19246, it was also reported as a project
authentication conflict
> - Agent verification also counts the refusal as a failed harness test,
and it parks the agent after three tries
> - This pull request gives the plugin a stable error code for a Runner
mismatch, and Core waits and retries when it gets that code
> - The benefit is that a release no longer blocks tasks or parks
agents, and a real authentication conflict keeps its own clear notice
## Linked Issues or Issue Description
Refs #19246 (it stopped reporting every precheck error as a project
authentication conflict)
Refs #19249 (the same bounded bootstrap retry for a stopped plugin
worker)
Refs #19302 (shares `environment-task-error.ts`; its diagnostic envelope
is unchanged)
Refs #19172 (related open PR: it reports an undescribable plugin Runner
as a retryable check in the same precheck)
Refs #18667 (a real `ai_connection_incompatible` conflict, which now
gets its own notice)
**What happened?**
A run started while its remote Runner was older than the app. The
environment plugin refused the task because the installed Runner did not
match the revision it expects. Core recorded a generic setup failure,
scheduled no retry, and blocked the task. Before #19246, Core reported
this as a project authentication conflict. An agent in `verifying`
counted the same refusal as a failed harness test. After three tries it
stopped verification until a person chose Retry.
**Expected behavior**
Core treats a Runner mismatch as temporary. A run retries without
blocking its task. A `verifying` agent checks again until the Runner
matches. Only a real `ai_connection_incompatible` result blocks the
task, with a notice that names the conflict.
**Steps to reproduce**
1. Use an agent on a plugin environment whose installed remote Runner
revision differs from the revision the plugin expects.
2. Wake the agent on a task, or put a new agent through verification.
3. The run fails with `setup_failed` and the task is blocked.
Verification parks after three tries.
**Paperclip version or commit**
master at
|
||
|
DottaandPaperclip
|
789a71bc17 |
fix(runner): retry quarantined tasks in their saved workspace (#19340)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native sessions retain execution ownership and action history across turns. > - Task Retry must prove that old execution has stopped before it starts another turn. > - An attempt can inherit its predecessor's identity and fail before it launches a provider. > - The cleanup proof searched under that failed attempt and compared unrelated event sequences. > - This pull request verifies the retained owner and starts one fresh authorized turn. > - Saved results and uncertain action outcomes remain intact. ## Linked Issues or Issue Description **What happened?** Task Retry accepted repeated clicks but kept the canonical request deferred with `executionWait.reason: local_cleanup`. It created no successor, although the retained local worker and provider had stopped. **Expected behavior:** After exact cleanup proof, admit one fresh turn. Preserve the old journal and result. Keep unresolved action outcomes unknown. **Steps to reproduce:** 1. Finish a local native turn with retained cleanup state and a saved result. 2. Let the next attempt inherit its identity and fail with `native_session_cleanup_quarantined` before any launch. 3. Upgrade across a tool-contract change, then click task Retry. 4. The old code requires the failed attempt's stop receipt and journal identity. It also requires the current tool fingerprint and equal sequence numbers across raw and normalized events. Those requirements cannot match the retained owner. **Paperclip version or commit:** Reproduced on `31bf6e4482`. **Deployment mode / installation method:** Authenticated, self-hosted source checkout. **Agent adapter(s) involved:** Native `paperclip_runner`, local Codex app-server. The shared physical-stop proof also covers ACPX. **Database mode:** Embedded PostgreSQL 18.1. **Access context:** Board task Retry. **Node.js version / operating system:** Incident: Node 24.20.0, Linux ARM64. Local verification: macOS ARM64. **Relevant logs or output:** Accepted Retry stays in `deferred_issue_execution`; no new run or provider launch occurs. **Relevant config:** Per-turn local native execution. The old tool contract differs from the current contract. **Additional context:** Related: #19311 prevents the oversized tool result that can lead to retained cleanup. #18217 covers broader cleanup isolation and retained-result maintenance. #19134 admits stopped Runner sources under the current run lease. This patch fixes explicit continuation admission on current master. ## What Changed - Resolve the retained journal's owner under the same company, task, agent, workspace, native session, and runner instance. - Require terminal ownership, settled controllers, released local leases, and exact inherited process birth. Refuse any launch evidence on the failed attempt. - Accept a current host's verified process absence from its exact launch receipt when an older server lacks a stop receipt. A later launch invalidates that observation. - Validate normalized receipts against their own IDs, sequences, and digests. Match ordered provider identities to the independent raw journal. - Keep tool compatibility at the session reuse boundary. Explicit continuation always uses a fresh session. - Bind the stop proof to its exact successor and canonical Retry receipt. Reuse the proven local workspace through normal preparation. Refuse changed bindings, closed or remote workspaces, pending operations, and other live leases. Projectless run-ID placeholders keep the existing fresh-workspace fallback. - Retain journals, results, files, and uncertain action outcomes. Add regression and opt-in live acceptance coverage. - Make the split-handoff browser test wait for both child-completion notices and all run settlement before teardown. Assert the final task set stays unchanged. ## Verification - Red: canonical Retry on the original code accepts one saved intent but creates zero successors and records `local_cleanup`. - Red: distinct raw and normalized event sequences reproduce that wait after owner resolution alone. After admission was repaired, live acceptance then exposed the empty-attempt workspace problem. A PostgreSQL regression also reproduced that wrong workspace selection. - Green at `6684df5e82`: **479 targeted cases**, including 26 new PostgreSQL cases for duplicate clicks, exact process and workspace ownership, controller/lease fences, changed state, and independent receipt integrity. The complete 1,150-test native executor suite passed earlier. - `pnpm -r typecheck` and `pnpm build` pass again at final head `432a52bea8`, including its build stamp. Token gates pass. Master now contains the unrelated directory-fixture type correction found during the first typecheck. - Real authenticated HTTP Retry with Codex `gpt-6.1-sol`, high effort, and per-turn execution passes again at final head `432a52bea8` (93.3 seconds). It preserves the first file line, appends the second once, starts a fresh session in the existing workspace, keeps the retained journal unchanged, commits finalization, and clears task locks. - The strengthened split-handoff browser scenario passes **three consecutive local runs**. Its original CI teardown and one same-head rerun failed when cleanup cancelled an active legacy completion-notification run and hit a PostgreSQL row-lock timeout. The scenario now waits for the complete handoff before cleanup. [Initial failure](https://github.com/paperclipai/paperclip/actions/runs/42106466518/job/127523693634), [same-head rerun](https://github.com/paperclipai/paperclip/actions/runs/42106466518/job/127531698321). These failures remain part of the verification record. - The local full `pnpm test:run` passed all six chat integration shards. Its obsolete pre-rebase attempt was stopped during the general server phase; no complete local-suite pass is claimed. Full CI, including both required gates, passed at `6684df5e82`. At final head `432a52bea8`, all initial CI groups passed except the unchanged status-card deadline test, whose post-wakeup clock assertion missed its bound by 10 ms. That exact case passed three unchanged local repetitions (16.8/27.0/18.3 seconds); the single isolated shard rerun passed all **2,218 tests** (123 files), with no source changes. [Initial timing failure](https://github.com/paperclipai/paperclip/actions/runs/42114473652/job/127550722187). - Greptile reviewed `6684df5e82` at **5/5**, with no actionable findings. A second reviewer identified the supported projectless placeholder case. A new real PostgreSQL regression reproduced its workspace-binding failure before the one-line fallback correction; all **45 focused cases** now pass, including default environment selection and changed binding/cwd refusal. Greptile reviewed final head `432a52bea8` at **5/5**, with no blocking findings. The second reviewer’s thread is resolved. - The patched Linux build verified the actual retained worker/provider ownership and process absence in a transaction that was rolled back. No retained state was retired or provider started during that pre-deployment check. ## Risks - This changes an execution admission proof. Incorrect ownership could permit a second executor. The proof retains process, controller, lease, scope, journal, and receipt fences, and duplicate admission is tested through the real queue. - Missing or conflicting evidence remains deferred. Windows, remote execution, automatic replacement, and session reuse retain their existing policies. - No database migration or UI change. The original affected task still needs deployment of the fix before recovery can be claimed. ## Model Used - OpenAI GPT-6 through Codex, with reasoning, repository tools, code execution, SSH diagnostics, and live provider testing. This session does not expose the exact backend model build or context-window size. The live test agent uses OpenAI `gpt-6.1-sol` at high effort. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Nicky LeachandPaperclip
|
2506bcdaf4 |
Admit exact stopped Runner sources under the current run lease (#19134)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native execution keeps provider state and a durable controller journal. > - A completed process can stop before the next run attaches. > - The current run must own recovery of that exact source. > - An old controller must lose permission to change the consumed source journal. > - This pull request adds explicit source admission and protected task dispatch. > - The benefit is a narrow recovery boundary that preserves the original successful result. ## Linked Issues or Issue Description Native prerequisites #19132 and #19153 are merged into master. This PR targets master and updates the TypeScript Runner driver contract alongside Core and SDK admission. This change adds the SDK and Core admission boundary. Executor selection and fresh source preparation are a separate change. Related #17916 supplies the workspace export holds preserved here. ## What Changed - Plugins advertise `codexSourceRecoveryVersion: 1` for Codex App Server and `acpxSourceRecoveryVersion: 1` for ACPX. Hosts can qualify either, both, or neither. Core selects the matching field, because a stopped-source check for one provider cannot establish the other provider’s recovery guarantees. ACPX orchestration is tracked separately in #19196. Use the existing exact source identity and `resumeFrom` under the authenticated current run. No public recovery mode is added. Require the claimed source, current account and prepared launch at protected dispatch. - Keep the first source binding pinned on the consumer lease, including after uncertain dispatch. A newly acquired coordinator can refresh only the same source. - Claim one stopped successful source under the current run's acquired native lease. Store the token-free consumer reference in existing lease metadata. Preserve the source status and result. - Check the exact six source identity fields, encrypted journal header, company, task, agent, environment, driver, workspace, account selection, and current coordinator generation. Keep workspace export and legacy settlement holds. - Serialize claims and checkpoint writers in the same coordinator lock order. Reject stale source writes after consumption. Check the current consumer before recovered source writes. - Check source admission before and after protected worker dispatch. Keep the request and lease owned by the current run while the native identity remains the source. - Allow the same owner to prepare B after a fresh source submission, using the exact consumed-source proof instead of a generic warm-reuse flag. Recheck that proof after dispatch. - Preserve ordinary cached adoption and repeated preparation after launch. The current authenticated owner can differ from the adopted PRP identity without requesting a new source launch. Keep ordinary suspended recovery and live task attachment checks. Extract the shared source resolver to avoid a module cycle, and make ordinary selection honor the same durable source consumer. ## Verification - Current head `cac9cf188a` includes merged #19300 and the symmetric capabilities introduced in `b7e25a64b4`. The task-owner and warm-rebind fixtures now identify their physical task. All 29 SDK schema cases, 95 Core ownership and source-admission cases, 25 native Codex and ACPX recovery cases, and the binary metadata check passed on this head. The SDK and Runner builds and server typecheck passed. Both native providers reject the other provider’s capability. GitHub CI is green and [standard Greptile scored this commit 5/5](https://github.com/paperclipai/paperclip/pull/19134#issuecomment-6645574114) with no open threads. - Previous head `bda49271a4ae362a15baf3fe6d6d297e25fd2000` is directly based on master after #19132 and #19153 landed. [GitHub workflow 42072192023](https://github.com/paperclipai/paperclip/actions/runs/42072192023) passed on its first attempt. All check runs completed successfully or as intentional workflow skips; Snyk passed. Standard Greptile scored this exact head 5/5. That review had no unresolved findings. - Runner and SDK TypeScript builds and server typecheck passed. The focused SDK, task-owner, task-runtime, PostgreSQL source and authority suites passed all 161 cases. The production transport-factory getter regression and 14 native-lane selection checks also passed. - The Linux native lane executed the new ordinary warm A→B process case. It kept one Runner/provider process and completed two turns on the same thread. The eager-getter mutation fails with `native_runner_source_attachment_pending`; preserving the live getter passes. The production factory test changes the current owner after transport creation and checks that same options object. - PostgreSQL cases cover both concurrent source-writer/claim orders, exact source submission under the current caller, source-switch denial after uncertain dispatch, same-source coordinator refresh, stale same-boot generation, corrupted journal headers, competing consumers, account changes and export holds. - The protected dispatch sequence submits A, prepares B and retries B preparation. It rejects undispatched sources, changed accounts and foreign target bindings. Source status and accepted result remain unchanged. - Ordinary adopted A preparation under current B, repeated ordinary preparation after launch and ordinary suspended continuation retain their existing path. Absent or false claims preserve ordinary identity denial; malformed claims fail closed. - Earlier development attempts exposed ordinary preparation regressions and the live-getter omission; the final regressions cover those fixes. The first embedded PostgreSQL attempt needed missing library aliases repaired in ignored dependencies. Those attempts are not passing evidence. - Historical, superseded CI heads had unchanged UI fake-timer and Playwright health-response-lifetime failures. Their isolated cases and scoped retries passed. The response-lifetime cause remains unproven. No UI/test timeout was changed. The final rebased-head workflow above passed without a retry. - Whitespace and added-line secret/PII scans passed. Full workspace checks ran on GitHub. No live vendor probe or deployment was performed. ## Risks - Any driver that used the earlier Codex TypeScript option must use `codexSourceRecoveryVersion`. The new plugin description rejects the generic spelling. The Launcher’s existing Codex-specific HTTP header can map to this property when the plugin adopts the new SDK. The change requires no journal migration or ROM change. - This is an inactive admission foundation. Executor orchestration is separate in #19157. Native restore integrity and provider stopped-source/account validation must be present before executor activation. The native prerequisite uses ordinary recovery followed by attachment. The composed executor proofs are in #19157. - The source claim is durable. A later controller must recover the same consumer or keep the hold. It cannot select another source after an ambiguous launch. - Admission adds bounded database checks around protected startup calls. Busy issue locks fail closed; other lock waits have a one-second bound. - No migration, snapshot format, proactive receipt, recurring worker, or deployment is included. Full workspace checks and platform coverage run in CI. These tests do not prove vendor history durability or a hosted rollout. ## Model Used OpenAI GPT-6.1 Sol (`gpt-6.1-sol`), high reasoning, through Codex with repository tools and code execution. The runtime does not expose a context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green on the current head - [x] Standard Greptile is 5/5 with no unresolved findings on the current head - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
22281d4b3a |
feat(task-chat): approve harness-proposed plans (Codex plan mode, Claude ExitPlanMode) (#19291)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents run through the native runner. Codex and Claude Code each have their own plan mode, and each produces a proposed-plan document. > - Paperclip already has a planning work mode, an issue `plan` document, and a `request_confirmation` approval gate for plans. > - These parts were not connected. The Codex proposed-plan item was transcript text only. The server path never put Codex into plan mode. Claude was never put into plan mode, and its ExitPlanMode plan was dropped. A planning run could close its task as done with no approval. > - This pull request turns a harness's own proposed plan into the issue's Plan document and the existing approval gate, for Codex and Claude. > - The benefit is that users approve or send back the agent's real plan from the task chat. Approve resumes the agent. Request changes sends the reason back to the agent. ## Linked Issues or Issue Description Refs #19252 **Subsystem affected** Native runner provider events (Codex and ACPX Claude), the runner Codex provider (Rust), server native plan synchronization, and the task chat UI. **Problem or motivation** Codex and Claude both produce a proposed-plan document in their native plan mode. Paperclip did not use it. In live tests on the server path, a Codex planning run was not in Codex plan mode, saved its own `implementation-plan` document, and marked the task done without approval. Claude planning runs were not in Claude plan mode, and the ExitPlanMode plan text never reached Paperclip. **Proposed solution** Add a canonical `plan.proposed` event for a harness's proposed plan. Put Codex and Claude into their native plan mode for planning runs. Save a completed proposal as the issue `plan` document with the existing `request_confirmation` gate. Agents can also propose a plan on a standard task; the work mode stays standard. **Alternatives considered** Reuse `plan.updated`. Its schema is the within-turn checklist and is not the durable Plan document, so a separate event keeps the two clear. Stronger prompt text without native plan mode. Live tests showed that the model kept its own document workflow. **Roadmap alignment** This reuses the issue plan document, planning work mode and approval gate that already exist. It adds one canonical provider event. ## What Changed - `packages/paperclip-runner/protocol/schemas/*`: add the `plan.proposed` event (`paperclip.plan.proposed.v1`: `proposalId`, `status`, `markdown` up to 60k characters, `source`). Regenerate the schema bundle, validators and manifest. - `packages/paperclip-runner/src/provider-events.ts`: map the Codex `plan` item and a `<proposed_plan>` block in a completed Codex message to `plan.proposed`. Add `claudePlanProposalEvent`. - `packages/paperclip-runner/runner/crates/runner-core/src/codex_provider.rs`: send Codex's native `collaborationMode: plan` on `turn/start` for a read-only (planning) Codex run. Before this change the server path never sent it. - `runner-core/src/acpx_event_payload.rs`: admit `plan.proposed` as ACPX rich display activity. - `drivers/acpx/codex-runtime-adapter.ts`: set Claude Code to native plan mode for a read-only run. Capture Claude's ExitPlanMode plan from its permission request, record it, and decline the native "Ready to code?" prompt. - `drivers/acpx/codex-acpx-driver.ts`, `cli/acpx-runtime-sidecar.ts`: emit the captured proposal as a rich event. Complete that turn instead of reporting it as cancelled. - `backends/codex-native-backend.ts`: per-harness plan-mode instructions, and a short opt-in hint for Claude on standard tasks. - `server/src/services/native-runtime/native-session-executor.ts`: synchronize a completed `plan.proposed` into the `plan` document and a `request_confirmation` in planning and standard mode. Do not change the issue status during the run (that revoked the run's lease). The run finalizer turns the pending confirmation into the review wait. - `server/src/services/heartbeat-run-summary.ts`: keep a `<proposed_plan>` block out of the posted reply. - UI: new `plan_proposal` provider family. The task chat shows a live "Writing plan" card while a proposal streams and a one-line record when it completes. Resolved plan reviews keep a receipt ("Approved Plan v1", "Requested changes to Plan v1: reason"). Plan previews skip section headings. - Storybook: `Tasks and conversations/.../Task chat/Proposed plan approval` shows the journey. - `doc/SPEC-implementation.md`: describe harness-proposed plan approval. ## Verification - `cargo test -p paperclip-runner-core --lib` (new: `read_only_codex_turns_use_native_plan_collaboration_mode`) - `cd packages/paperclip-runner && npx vitest run src/provider-events.test.ts src/drivers/acpx/codex-runtime-adapter.test.ts` - `cd server && npx vitest run src/__tests__/native-plan-proposal-sync.test.ts src/__tests__/heartbeat-run-summary.test.ts` - `cd ui && npx vitest run src/components/transcript/native-run-events.test.ts src/components/task-chat` - Live test on a local worktree instance with real harnesses: - Codex (`gpt-5.6-sol`), planning task: Codex asked one question, then produced a native plan item. The task showed Plan v1 with Approve plan / Request changes and moved to in review. Approve changed the work mode to standard. Codex then implemented the plan and its tests passed. - Codex, Request changes with a reason: Codex woke in planning mode and produced Plan v2 that includes the change. - Claude (`claude-opus-5-5`), planning task: Claude was in plan mode. ExitPlanMode became Plan v1 and the turn completed. Approve resumed Claude in standard mode, and Claude implemented the plan. - Claude, standard task that asks for a plan first: Claude stopped before code changes and asked for plan approval. The work mode stayed standard. - Storybook: open `Proposed plan approval` and check each stage. ## Risks - Codex planning runs now really use Codex plan mode on the server path. Codex can ask a clarifying question before it proposes. This matches Codex plan mode. - Claude planning runs now use Claude plan mode. A declined ExitPlanMode ends the Claude turn; the runner reports it as completed. Claude's tool row for that request records a rejection from Claude's side. - Codex does not stream the proposed plan on the server path yet, so the live card shows a placeholder until the plan completes. - A `<proposed_plan>` block in any completed Codex message on a standard task now asks for approval. This is the documented opt-in path. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file edits, browser). Long context. No extended thinking mode was used. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
b86f55f0ba |
test(server): remove two more server test races (#19330)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The server Vitest shards gate every pull request > - Two more server tests failed on pull requests that do not touch them (#19309, #19265) > - #19314 fixed four other flakes. These two have different causes > - Random CI failures slow every contributor and hide real regressions > - This pull request removes both root causes without retries or skips > - The benefit is fewer false CI failures ## Linked Issues or Issue Description Refs #19309, #19265, #19314 **What happened?** - #19309, job 127459915333: `workspace-recovery.test.ts` > "keeps ancestor Retry held when current proof changes: sibling_hold" received `requestId: null` from a second Retry request. - #19265, job 127481105563: `paperclip-control-plane-port.test.ts` > "retains a durably recorded governed terminal while its postcommit observer is pending: checkpoint timeout" failed with "native session checkpoint timed out after 100ms" before the terminal was recorded. **Root causes** 1. Workspace recovery (test race). `POST /issues/:id/retry` returns 202 and then starts `heartbeat.resumeTaskRetry` without waiting. The proof-change tests change state after the receipt. When the background nudge runs first, it admits the Retry and queues a run before the change. The new queued run is then the latest run, so the second Retry in `sibling_hold` is stale. I proved this locally: a 1.5 s pause after the receipt made every gate in the family fail with a queued run in the issue history. 2. Control-plane checkpoint timeout (budget too tight). `checkpointTimeoutMs: 100` applies to every checkpoint in the session, including the normal checkpoints before the terminal. On a loaded runner, one of those took more than 100 ms. The test expects only the checkpoint that the pending observer blocks to time out. ## What Changed - `server/src/services/workspace-recovery.test.ts`: add `receiptOnlyRecoveryApp`. It builds the same route app, but the route nudge does nothing. The two proof-change families ("keeps ancestor Retry held…" and "holds a failed-preparation HTTP Retry…") use it and call `resumeTaskRetry` explicitly, as before. - `server/src/services/native-runtime/paperclip-control-plane-port.test.ts`: the checkpoint budget is 1 s. The timeout case now waits for the run to fail with the timeout while the observer is still blocked, instead of a fixed 700 ms sleep. This also proves that the timeout fires while the observer is pending. ## Verification - With the forced 1.5 s pause and the fix applied: both workspace-recovery families pass (80 passed). - `npx vitest run src/services/workspace-recovery.test.ts -t "keeps ancestor Retry held when current proof changes|holds a failed-preparation HTTP Retry when current proof changes"`: 80 passed. - `npx vitest run src/services/native-runtime/paperclip-control-plane-port.test.ts`: 94 passed, run 3 times. ## Risks - Low risk. The changes are only in tests. The route nudge is a latency hint ("Maintenance redelivers it"). Other tests still cover it. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edit, and GitHub CLI). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
ffd9591ab4 |
fix(recovery): preserve cancelled files after a reusable sandbox stops (#19305)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run writes its task files in a remote sandbox. The task
workspace records that run as its writer until the files are saved.
> - When a user stops or reassigns a run, the cancelled-workspace
preservation worker records the unsaved files as file recovery work.
> - A warm run (`keep_running`) that is not successful releases its
reusable sandbox through stop-and-retain. That release can finish before
the worker sees a terminal coordinator.
> - The worker accepted only an active lease or a released ephemeral
lease. It ignored the released reusable lease, so it never recorded the
file recovery.
> - The workspace kept the ended writer's hold. Each successor run
waited, then failed with "The workspace still has an unfinished writer
or save operation" after the one-hour cutoff.
> - This pull request accepts a released reusable lease that has a
confirmed stop-and-retain receipt as a preservation source.
> - The benefit is that the stopped run's files become recovery work,
and the task is not stranded behind a hold that nothing settles.
## Linked Issues or Issue Description
Refs #18820 (preserves files when Stop finds no live controller). Refs
#17677 (schedules cancelled workspace preservation). Refs #19285 (parks
a workspace wait behind an ended writer).
**What happened?**
A board user reassigned a task during a warm native run. Stop cancelled
the run with `issue_reassigned`, and the Stop acknowledgement was
recorded. The executor settled the coordinator to `terminal_failure`
with no owner. About eight seconds later, terminal release stopped and
retained the reusable Daytona sandbox. The lease became `released` with
`leasePolicy: reuse_by_environment`, a `stopped` termination receipt and
a stop-and-retain receipt. The task workspace stayed `dirty` with
`writerRunId` set to the cancelled run. No file recovery was recorded.
The new assignee's runs waited as `workspace_busy` for one hour. Then
each run failed `setup_failed`, and the task became blocked.
The heartbeat run row showed `nativePhase: "observed"`. That value is
stale: the checkpoint persist wrote it during teardown. The coordinator
itself was `terminal_failure`: the run's execution projection was
`recovery_needed` with no matching recovery action.
**Expected behavior**
The preservation worker records the stopped run's prepared files as file
recovery work. The lease keeps its release status and receipts. No
second stop and no new provider turn start.
**Steps to reproduce**
1. Cancel a native run with an acknowledged run-only Stop. Its remote
workspace reference is `prepared`, and the coordinator is
`terminal_failure` with no owner.
2. Release its reusable lease through stop-and-retain (`released`,
`reuse_by_environment`, confirmed stop receipt).
3. Run the cancelled-workspace preservation worker or the startup scan.
4. On master, the run has no `workspaceRestoreRecovery`, and no recovery
action exists.
## What Changed
-
`server/src/services/native-runtime/native-failed-workspace-recovery.ts`:
add one source check, `cancelledWorkspaceSource`, for cancelled
preservation. The worker check and the startup scan both use it. It
keeps the active and ephemeral cases. It adds a released reusable lease
only when the stop-and-retain receipt is confirmed
(`hasConfirmedSandboxStopAndRetain`) and `preparedRemoteWorkspaceSource`
proves the exact remote-workspace binding. This is the same proof the
restore recorder uses. Native-sync runs have no such proof and stay
ineligible. An ineligible source causes no run write or notification.
- The restore recorder (`preserveLegacyWorkspaceRestoreSources`) already
accepts this source. It checks the confirmed receipt and the exact
prepared workspace binding, and it keeps the lease status, policy and
receipts. This pull request does not change it.
- `native-cancelled-workspace-preservation.test.ts`: add the incident
shape as a regression for the worker and the startup scan, and add
safety cases.
- `doc/execution-semantics.md`: document the stopped reusable source and
the writer hold until recovery.
## Verification
From `server`:
```sh
npx vitest run src/services/native-runtime/native-cancelled-workspace-preservation.test.ts
```
- Red before the fix: the two new regression cases failed (`expected {
nativeCancellation: … } to match object { …, workspaceRestoreRecovery: …
}`). 30 other cases passed.
- Green after the fix: 36 passed.
- Review follow-up: the cases `invalid-receipt`, `expired-lease` and
`native-sync` failed on the previous head
|
||
|
DottaandClaude Opus 5.5
|
dd889e6a0b |
feat: show the assignee's provider usage limit in task properties (#19270)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents run on the native Runner through the Codex app-server and the Claude ACP harness > - Both harnesses report the provider account's rate-limit windows: Codex sends `account/rateLimits/updated`, and Claude sends its SDK `rate_limit_event` as `usage_update._meta["_claude/rateLimit"]` > - Paperclip dropped both signals, so an operator could not see that an agent was near or past its provider limit > - This pull request normalizes both signals into one account-scoped snapshot, keeps the latest snapshot per agent, and shows it as one "Usage limit" line in the task properties pane > - The benefit is that an operator sees "5h 78% · resets 15:40" (amber above 80%, "Limited until …" when the provider refuses work, "using credits" when a used-up window falls back to credits) next to the assignee, and can click it for every window ## Linked Issues or Issue Description Refs #19246 Refs #19259 **Is your feature request related to a problem?** Codex and Claude report provider rate-limit windows during a run. Paperclip dropped them: the Codex notification fell through the Rust normalizer with no branch, and ACPX dropped Claude's `_meta`. Operators could not see how close an agent was to its provider limit. **Describe the solution you'd like** Store the latest reported windows per agent and show one compact line in the task properties pane, with a popover that lists every window. **Describe alternatives you've considered** A full gauge in the agent list and the run header. A design review chose one property line with a details popover instead. ## What Changed - Runner (Rust, the runnerd path): new `provider_rate_limits.rs`. Codex `account/rateLimits/updated` becomes a `provider.notice.recorded` event (category `provider.rate_limits`, scope `account`). `CodexProviderState` merges sparse updates and emits a notice only when the windows change. ACPX `usage_update` with a sanitized `rateLimit` field also emits the notice. - Runner (sidecar): `acpx-runtime-sidecar.ts` forwards only `status`, `rateLimitType`, `utilization` and `resetsAt` from `_meta["_claude/rateLimit"]`. This needs the ACPX `_meta` passthrough from #19259. Until that merges, Claude emits nothing. - Runner (TypeScript in-process drivers): the same mapping in `codex-session-notifications.ts` and `codex-acpx-driver.ts` through `drivers/provider-rate-limits.ts`. - Shared: `provider-rate-limits.ts` holds the snapshot type, the parser, the merge, and `summarizeProviderRateLimits` (fullest current window, expiry by reset time, limited versus using credits). - Server: `agent-provider-rate-limits.ts` merges each notice into `agent_runtime_state.state_json.providerRateLimits`, scoped by company and agent. A session reset keeps the snapshot. No migration. - UI: `AgentRateLimitPropertyValue` adds a "Usage limit" row under the assignee in `IssueProperties`. Transcripts treat the notice as run-log only. A run status event refreshes the assignee's runtime state. - Storybook: `Tasks and conversations/Task creation and lifecycle/Usage limit property`. - Docs: `doc/run-log-events.md` describes the notice and its storage. ## Verification - `cd packages/paperclip-runner/runner && cargo test -p paperclip-runner-core --lib rate_limit` and `--test acpx_provider_events`: sparse Codex merge, dedupe, reached window, checkpoint round trip, Claude utilization and rejection, notice shape. - `cd packages/paperclip-runner && npx vitest run src/drivers/provider-rate-limits.test.ts src/drivers/codex` - `cd packages/shared && npx vitest run src/provider-rate-limits.test.ts`: summary, expiry, limited, using credits, parse and merge. - `cd server && npx vitest run src/__tests__/agent-provider-rate-limits.test.ts`: create and merge, out-of-order snapshot, company isolation, unrelated notices, session reset keeps the snapshot. - `cd ui && npx vitest run src/components/transcript src/components/issue-properties src/context/LiveUpdatesProvider` - `pnpm -r typecheck` and `pnpm check:token-gates` pass. - Live test on a local worktree instance: a native Codex agent on a ChatGPT subscription login ran a real task. The runner recorded one `provider.rate_limits` notice. The server stored it. The task properties pane showed "7d 100% · using credits" (this account's weekly window was used up and had credits, and the run succeeded). The transcript showed no extra row. - Not verified live: Claude. The local Claude CLI was not signed in, and the Claude path also needs #19259. The Rust and sidecar paths have tests. - Not run locally: the full `pnpm test:run` and `pnpm build`. CI runs them. ## Risks - Low risk. The snapshot is display data. It does not hold or start work. The existing provider-limit wait (#19246) still reads typed run failures. - Codex can send a rate-limit update often. The runner emits a notice only when the windows change, so run-log growth stays small. - With API-key auth, the harnesses report no windows, so no row appears. This is the intended result. - `CodexProviderState` gains two optional serde fields with defaults, so older checkpoints still load. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, file edits, browser) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Nicky LeachandPaperclip
|
31bf6e4482 |
Pin plugin Runner cleanup to its physical task (#19300)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Plugin Runner owners manage a physical task through an environment lease. > - The existing replacement path can assign a new physical task to that same lease. > - A delayed old owner can then stop the replacement or mark it stopped. > - Cleanup must keep the original physical task identity across RPC and database writes. > - This pull request pins that identity and keeps cleanup available to the new owner. ## Linked Issues or Issue Description Refs #19134 and #19157. This fix is based on master and does not enable repeated stopped-source recovery. A search found no duplicate fix. **What happened?** The maintained `createOwnedSandbox` replacement path changes `providerLeaseId` on one lease. An old task owner uses that mutable lease for a later Stop. A stopped-state response already in flight can also update the new task's flags. **Expected behavior** The old owner must address its original task. Its late cleanup writes must not alter the replacement. The replacement's own owner must still stop and complete it. **Steps to reproduce** 1. Create a plugin task owner for a stopped task. 2. Replace that task through the existing describe/startup path on the same lease. 3. Resume an old checkpoint or a held old status response. 4. Observe a Stop for the replacement or an incorrect stopped flag before this fix. ## What Changed - Require each task owner to keep the physical ID returned by its description. Reject a rebind to a different task. - Carry the ID through the existing internal task context after startup ends. Reject a changed physical task before RPC dispatch. - Match the physical task in stopped-state, completion and credential-handoff writes. - Hold the exact lease row through the retained-session database update and local checkpoint save. - Cover the maintained replacement path with real PostgreSQL tests. Preserve positive cleanup for the replacement owner. Document this boundary. ## Verification - Both replacement regressions failed on unmodified master `10a7d039c8`: delayed old-owner Stop targeted the replacement, and a held old status response marked it stopped. - Six focused files passed 160 tests: task authority, task runtime, warm tasks, execution target, credential writeback and task owner. - Four selected executor factory cases passed. They cover task-only providers and the physical task binding after the startup context ends. - Runner TypeScript, plugin SDK and adapter-utils builds passed. Server TypeScript check passed. Diff and added-line secret scans passed. - All 15 setup-probe cases passed with the actual Runner and fake provider binaries. The first local attempt lacked cargo on PATH. A sandboxed native attempt timed out in seven process cases; the unchanged run with loopback permission passed all 15. A separate initial DB integration attempt was denied a loopback listener; its permission-correct run passed. - Review found three missed constructors in the existing SDK/Core lifecycle fixture. They now use the actual describe result. All four cases in that real PostgreSQL and worker RPC file passed. - The first GitHub attempt failed in the unchanged GitHub recovery pagination case in `chat-channels.integration.test.ts`: its zero-request assertion observed a configuration GET. That exact case passed locally without code or assertion changes. The one failed-job-only retry passed; the initial failure log is retained. - [GitHub workflow 42082587576](https://github.com/paperclipai/paperclip/actions/runs/42082587576) passed on its second attempt. All CI checks completed, and Snyk passed. Standard Greptile scored exact head `fb914d98b99e` 5/5 with no unresolved threads. The branch is mergeable and ready for review. ## Risks The task context change is internal. It adds no plugin wire field, schema or Runner mode. A stale owner now gets an error and cannot clean up a replacement. A new owner can still clean up that task. The retained checkpoint write holds one existing lease row during a bounded local file write. Repeated stopped-source incarnation admission, runtime pins and deployments are outside this change. ## Model Used OpenAI GPT-6.1 Sol (`gpt-6.1-sol`), high reasoning. Used repository tools, TypeScript and real PostgreSQL tests. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used with version and capability details - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in-PR using the bug template - [x] I have not referenced internal or instance-local Paperclip issues or links - [x] My branch name describes the change - [x] I have run the focused tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation - [x] I have considered and documented risks - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no unresolved findings - [x] I will address reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
8464a8c9ae |
fix(native): settle a bootstrap retry that lost its task lock (#19307)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native run that fails before any provider work gets a bounded same-run bootstrap retry. Restart recovery decides whether that retry can run. > - The retry needs the task's issue lock. A user comment or another wake can move that lock to a new run before the retry is due. > - Restart recovery then returned `issue_execution_lock_changed` without a write. The same check ran on every recovery pass and never ended. The coordinator stayed `retryable_failure`. > - The orphaned-lease sweep waits for a terminal coordinator before it preserves the workspace. So the sandbox lease stayed active, and every later run for the task was cancelled as `workspace_busy`. > - This pull request gives a lock-lost bootstrap retry the same bounded admission that a superseded remote retry already has. > - The benefit is that the retry settles, the board gets a reconciliation action, and the sandbox lease is released instead of blocking the task. ## Linked Issues or Issue Description No public issue exists. Bug description: **What happened?** On a hosted instance, a native agent run failed at startup because the provider rejected `session.open` with a rate limit. Paperclip scheduled a same-run bootstrap retry. Seconds later, a new run for the task took its issue lock, most likely woken by a user comment. From then on, every restart recovery pass for the old run returned `blocked` / `issue_execution_lock_changed` and wrote nothing. The run recorded about 1,000 recovery events over 12 hours. Its coordinator stayed `retryable_failure` with `recoveryMode: bootstrap_retry`. The scoped orphaned-lease sweep found a pending workspace export for the lease. It could not preserve the workspace, because `preserveNativeFailedWorkspace` requires `terminal_failure`. So the lease stayed `active`, and 16 later runs for the task were cancelled as `workspace_busy`. Three more runs on the same instance were stuck in the same state. A board Stop on the failed run moved the coordinator to `terminal_failure`, and the sweep then released the lease within about a minute. **Expected behavior** A bootstrap retry that cannot get its task lock back settles within a bounded number of recovery passes. The coordinator reaches `terminal_failure`, the board gets a reconciliation action, and the existing orphaned-lease sweep can preserve the workspace and release the sandbox lease. **Steps to reproduce** 1. Fail a native remote run before provider work, so its coordinator is `retryable_failure` with `recoveryMode: bootstrap_retry`. 2. Move the task's `executionRunId` to a newer run for the same task, or clear it. 3. Run `claimNativeRestartRecoveries` for the failed run. 4. Before this change, it returns `blocked` with no `retryAt`, and the coordinator stays `retryable_failure` on every pass. The new tests in `native-remote-retry-admission.test.ts` reproduce this. **Paperclip version or commit** Observed on a hosted deployment. Reproduced in tests on `master` at `10a7d039c8e`. Related: #19190 (a `session.open` model rejection no longer schedules a doomed bootstrap retry), #19131 (preserve superseded observed workspaces) and #19281 (release the leases of a cancelled legacy run). None of them changes this admission path. ## What Changed - **Restart recovery (`server/src/services/native-runtime/native-restart-recovery.ts`)**: a new condition, `lockLostBootstrapRetry`, matches a failed run whose coordinator is `retryable_failure` with `recoveryMode: bootstrap_retry` and whose issue lock is not on the run. - A newer run for the task: the existing superseded check now covers this case too. It defers with `native_retry_source_superseded` as permanent, so admission is exhausted at once. - No newer run (the lock was released): recovery defers with `issue_execution_lock_changed` through the existing `deferRemoteRetry`. That path persists a deadline, backs off and exhausts after the system recovery budget. - When admission is exhausted, the existing hold path writes `terminal_failure` / `native_restart_recovery_blocked`, keeps the provider evidence and creates the board reconciliation action. This PR adds no new terminal path. - A bootstrap retry that still holds its own lock is unchanged. The exact-checkpoint remote retry path is unchanged. - **Docs (`doc/execution-semantics.md`)**: the bootstrap retry section now states the lock-lost rule. ## Verification - New test `settles a %s bootstrap retry that lost its task lock instead of re-checking forever` in `server/src/services/native-runtime/native-remote-retry-admission.test.ts`, for the `superseded` and `released` cases. The fixture is a genuine pre-provider failure: a `session.open` rejection with no session checkpoint, no runner checkpoint row and no provider evidence. - On `master` without the fix, both cases fail: recovery returns `blocked` without a deadline, and the coordinator stays `retryable_failure`. - With the fix, `superseded` settles on the first pass, and `released` settles after `SYSTEM_RECOVERY_MAX_RETRIES` deferred passes. Each deferred pass persists its `nextAttemptAt`. Both end in `terminal_failure` with one board recovery action, and later passes open no transaction. - `npx vitest run src/services/native-runtime/native-remote-retry-admission.test.ts` (from `server/`): 92 passed. - Related suites, each run on its own with exit code 0: `native-ingress-bootstrap-retry`, `native-startup-retry`, `native-scheduled-resume-lock`, `native-restart-recovery-isolation`, `native-runner-restart-recovery`, `native-runner-restart-preparation`, `native-runner-restart-diskless`, `native-warm-transition-restart`, `native-controller-recovery`, `native-failed-workspace-recovery`, `native-session-resumption`, `native-startup-cleanup-recovery`, `heartbeat-observed-workspace-preservation`, `heartbeat/recovery`, `runtime-selection`, `remote-workspace-migration` and `native-session-executor` (1,149 tests). Two other suites each had one local failure that this change does not cause: - `heartbeat-process-recovery`: "redacts opaque environment-bound credentials from Sentry diagnostics" fails the same way on this machine without the change. - `chat-channels.integration`: "settles a shared-endpoint accepted-response backlog without retrying its own locks" failed once under full-suite load. It passes alone, with and without the change. - `pnpm run typecheck` in `server/`: passed. ## Risks - Low to medium. The change only affects failed bootstrap retries whose lock is not on the run. Before this change, such a retry could never run, so no working path is removed. - A released lock that comes back to the same run during the backoff still resumes normally, because the condition reads the current lock on each pass. - When the retry settles, the task gets a board reconciliation action. Before this change, the task was silently stuck behind `workspace_busy`. - No migration and no API change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, code search, test runs), extended thinking, and a read-only investigation subagent for the root cause. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
74b11014c6 |
fix(server): stop a queued answer from blocking saved-work recovery (#19318)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native run can accept its result before the workspace copy-back finishes. If a server restart interrupts the copy-back, the run keeps an owner receipt for the server that stopped. > - Another server takes over only after the cloud platform proves that the old deployment stopped. After that proof, recovery checks that the task has not moved on, so old files never overwrite newer work. > - That check counted any queued run on the task as newer work. > - The user's answer to the run's question queues the assignee's next wake. That wake cannot claim while the saved run holds the task's execution lock, so it waits for the copy-back. > - Recovery waited for the wake and the wake waited for recovery. The task stayed stuck, and nothing was logged. > - This pull request ignores a later wake for the same assignee that has never started, and logs waits that the worker stops retrying. > - The benefit is that answering a question during an interrupted copy-back no longer strands the task. Every other newer-work check still applies. ## Linked Issues or Issue Description Refs #15824 (added platform-verified takeover and these binding checks) and #19111 (worker concurrency and ordering for the same copy-backs). This pull request keeps the rest of #15824's fence unchanged. **What happened?** On a hosted instance, a server restart interrupted a copy-back after the worker had written its owner receipt. The run had asked the user a question, and the user answered. The answer's wake was queued. Every later server got a valid stop proof for the old deployment and then got `binding_changed` from the shared binding check: the only failing condition was the queued answer wake. The worker treats `binding_changed` as a final wait and returns without logging. The run stayed `running` and the wake stayed `queued` for about 19 hours, across many restarts. **Expected behavior** A wake that is waiting for the saved run does not stop the saved run's recovery. Runs that could change the workspace still prevent takeover, and the publication check still stops a copy-back if such a run appears. **Steps to reproduce** 1. Accept a native result and leave a copy-back owner receipt from a stopped deployment. 2. Queue the assignee's next wake on the same task, for example by answering the run's question. 3. Let the reconciler or the workspace worker revisit the saved result with a verified stop proof. On `master` the takeover returns `binding_changed` and nothing is copied or committed. The new database tests run these steps against PostgreSQL. **Paperclip version or commit** `master` at `03eb49053fe`. **Deployment mode** Cloud-managed instances with platform stop verification. The same binding check also guards operator owner recovery, so self-hosted operator recovery had the same deadlock. ## What Changed - `server/src/services/native-runtime/native-workspace-recovery-binding.ts`: the newer-run check ignores a run when all of these hold: - It is `queued` and has never started (`startedAt` is null). - It belongs to the run's agent, and that agent is the task's assignee. - It was created after the saved run. - Its wake reason is not `source_scoped_recovery_action`. The run claim blocks such a run while the saved run holds the issue's execution lock (`lockIssueExecutionClaim` in `heartbeat/queue.ts`). Runs that have started, other agents' runs, recovery-action wakes and older queued runs still prevent takeover. - `server/src/services/native-runtime/native-workspace-finalization-work.ts`: log `runId` and `reason` when the worker stops retrying a wait (`binding_changed`, `owner_unverified`). It does not log `owner_active`. - `doc/native-workspace-finalization-recovery.md`: describe the waiting-wake rule in the shared binding section. - `server/src/services/native-runtime/native-workspace-queued-wake-recovery.test.ts` (new): the queued-wake tests, with a small fixture of their own. They are in a separate file so `native-workspace-finalizer.test.ts` stays under 1,000 lines. ## Verification - Failing first. These tests in `native-workspace-queued-wake-recovery.test.ts` fail against the original check and pass with this change: - Operator owner recovery with an answer wake queued. - Automatic takeover with the wake still queued. - Automatic takeover where the wake starts during copy-back. On `master` takeover is refused before this point; with the change, publication is stopped. - The real reconciler, from `retryable_failure`, with a stopped-deployment owner and a queued answer wake. It now copies once, commits the saved result and leaves the wake queued with its claim no longer blocked. - The fence is kept: three new cases (a requeued wake that already started, a recovery-action wake, another agent's queued run) still get `binding_changed`. They pass both before and after the change. - Changed test: "does not reopen an old result after a newer task execution exists" now inserts a started run instead of a never-started queued run of the same assignee. The queued case is now the new operator-recovery test above. - `cd server && npx vitest run src/services/native-runtime/native-workspace-queued-wake-recovery.test.ts src/services/native-runtime/native-workspace-finalizer.test.ts`: 2 files, 72 passed. With the original binding file restored, the 4 positive queued-wake tests failed and the 3 fenced cases passed. - Related suites (finalization work, platform proof, export retry, finalizer failure, recovery actions, owner-recovery routes, status reassessment, scoped deadline work): 8 files, 499 passed. - After rebasing onto `03eb49053fe`: finalizer and finalization-work suites, 97 passed. - `cd server && npx tsc --noEmit -p tsconfig.json`: pass. - Not verified on a live instance yet. The diagnosis comes from that instance's run, finalization, workspace-operation and recovery-action rows, plus a direct stop-proof request that returned a valid receipt. ## Risks - A copy-back can now finish while the assignee's wake is queued. The same-process retry path already does this, because it does not run the binding check. The takeover path now matches it for wakes that cannot start yet. - If some path starts such a wake despite the execution lock, `assertHeld` sees a running newer run and stops before publication. The window between that check and the write is the same as for the other binding conditions. - No schema migration. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) through Claude Code, with tool use (shell, file edits, test runs, read-only database and log queries). The runtime does not expose the context window or reasoning configuration. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
55389554c2 |
fix(runner): paginate workspace discovery before journal persistence (#19311)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native agents discover reusable workspaces through `list_workspaces`. > - The tool returned every authorized workspace in one result. > - A large inventory exceeded the durable command journal limit. The result stayed pending, and session cleanup blocked the next turn. > - This pull request limits discovery in SQL and returns explicit continuation metadata. > - Agents can inspect every authorized workspace without sending the full inventory in one command. ## Linked Issues or Issue Description Related: #15749 introduced the task workspace tools. No duplicate pagination fix was found. **What happened?** A native workspace discovery call returned a multi-megabyte inventory. Persisting its result failed with `command_payload_too_large`. The preceding answer could still be saved, but cleanup retained the unsettled tool call. A later turn failed before provider work with `native_session_cleanup_quarantined`. **Expected behavior** Workspace discovery must return a manageable page. Agents must be able to request the remaining authorized rows. Existing visibility checks and journal integrity checks must stay effective. **Steps to reproduce** 1. Create 2,000 task-selectable workspaces with ordinary repository paths. 2. Call `list_workspaces` from an admitted native run. 3. On the old code, the real API regression returns 869,781 bytes. The command journal allows about 510 KiB. 4. On the repaired code, follow `nextCursor` to inspect all rows. Pages contain at most 100 rows and fit within 128 KiB. **Paperclip version or commit** Reproduced on source commit `7351a9d74f` before the repair. **Deployment mode** Isolated source checkout with real PostgreSQL and authenticated native agent APIs. Hosted deployment and incident recovery remain separate. ## What Changed - Validate workspace list limits and UUID cursors with a JSON Schema pattern supported by strict validators. Return `workspaces`, `limit`, `returnedCount`, `hasMore`, and `nextCursor`. - Apply company, visibility, lifecycle, and cursor conditions before the SQL limit. Order pages by workspace ID. Fetch one extra row to detect continuation without a count query. - Fit encoded pages within 128 KiB, including JSON escaping and Unicode. If one summary cannot fit, return explicit string previews with `truncatedFields`. The authorized detail API retains its full record. - Keep existing unpaged summary callers on their array response and recent-use order. - Update the native tool descriptor, generated contracts, catalog fixture, and execution documentation. - Add regression coverage for large inventories, complete pagination, stable pages, private and foreign rows, invalid inputs, and legacy summary behavior. Add an opt-in two-turn live Codex acceptance check. ## Verification - Red before repair: `task-workspace-api.test.ts -t "pages a large authorized workspace inventory"` failed because the result was 869,781 bytes instead of below 128 KiB. The transport regression also failed at 2,284,671 bytes. - Green after repair and review changes: 120 tests passed across the workspace API, transport, routes, native authority, coordinator, historical-question, and Codex vertical-slice suites. They cover complete discovery, byte-sized pages, oversized individual summaries, full detail retrieval, private/foreign/lifecycle exclusions, and strict schema compatibility. - Journal safety: two focused durable control-plane tests passed. Oversized result rejection and lossless valid result delivery remain effective. - Tool catalog and reconciliation: 30 tests passed. - `pnpm -r typecheck`: passed. - `pnpm build`: passed again on final commit `bc10880da6`. Server typecheck also passed after the review changes. The original affected suites passed on the rebased first commit (31 tests), and the expanded review regression set passed afterward (120 tests). - Real provider acceptance passed again on final commit `bc10880da6`: local macOS arm64, rebuilt native Runner, `gpt-6.1-sol`, high effort, per-turn lifecycle, and 2,000 persisted workspaces. Both turns succeeded. Persisted call inputs and results prove that the second page uses the first result’s cursor, both calls succeed, their IDs do not overlap, and the encoded results stay within 128 KiB. The second turn kept the provider conversation and file. Both task locks were clear after completion, and no recovered execution failure was recorded. Duration: 166.92 seconds. - Full local `pnpm test:run` stopped in the chat shard: 330 tests passed and one Slack registration test failed with `socket hang up`. That exact test passed in isolation without changes, and the final-head CI chat shards passed. A broader local managed-Slack follow-up produced no result after about eight minutes and was interrupted; it is not counted as a pass. The original full-suite failure remains recorded. - The first CI head failed native tool-schema compilation because strict validators did not recognize `format: uuid`. The revised descriptor uses a UUID pattern. The previously failing coordinator, native vertical-slice, authority, and history suites now pass locally. The stronger live cursor-receipt acceptance check passed. Final-head CI completed on `bc10880da6`: 57 checks passed and five optional checks were skipped, with no failures or pending checks. Greptile reviewed that commit at 5/5; both prior findings are resolved. - Dependency cost: paged discovery runs the existing visibility-scoped summary query with an ID cursor and at most 101 returned rows. It adds no count query, filesystem scan, provider call, or background worker. ## Risks - Native `list_workspaces` now returns a page object instead of an array. Individual summaries can be explicit previews when a single record exceeds the byte budget. Its advertised descriptor and generated contracts change together. Agents must follow `nextCursor` until null for a complete inventory. - Each page rechecks current visibility. Concurrent insertions, deletions, or access changes require a fresh scan when a complete point-in-time inventory is needed. - No migration or journal-limit change. An existing cleanup quarantine still requires normal recovery after deployment; this PR does not bypass it. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, shell tools, code editing, and test execution. The exact served model revision and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
5de779848b |
fix(recovery): save a reassignment-stopped writer's files from its stopped reusable sandbox (#19304)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A remote task workspace stays held by its last writer run until that writer's files are saved > - A reassignment stops the old owner's run. The stopped writer still holds the workspace with unsaved files. > - When the run used a reusable (warm) sandbox, the stop leaves its lease `released` under the reusable policy, with a stop-and-retain receipt > - Cancelled-workspace preservation only admitted `active` or `ephemeral` sources. It never recorded the save for this stopped writer, so no workspace recovery ran, and the hold stayed forever > - The new owner waited, then failed setup after the one-hour cutoff with "The workspace still has an unfinished writer or save operation" > - This pull request admits the stopped reusable source, so Paperclip saves the stopped writer's files and releases the hold. A waiter keeps waiting while that save is in progress. > - The benefit is that a reassigned task continues with the old owner's saved work, and the new owner does not fail setup ## Linked Issues or Issue Description Refs #19285 (parks a waiter behind an ended writer). Refs #19074, #18820. **What happened?** A board user reassigned a task during a native run on a reusable sandbox. The old run was cancelled with `issue_reassigned`. Its sandbox was stopped and its lease was released under `reuse_by_environment`. The workspace still named the stopped run as its dirty writer. No save operation and no workspace recovery action were ever recorded for that run. The new owner's runs were cancelled `workspace_busy` repeatedly. After more than an hour, a new run failed setup: "The workspace still has an unfinished writer or save operation." **Expected behavior** Paperclip saves the stopped writer's files from its stopped sandbox within a bounded time and releases the hold. The new owner waits for that save and then continues. If the save cannot finish, its retries end with a named board recovery action. **Steps to reproduce** 1. Give a native run a prepared remote workspace on a lease with policy `reuse_by_environment`. 2. Stop the run (for example by reassignment), so the lease is stopped and released with a stop-and-retain receipt. 3. Run cancelled-workspace preservation. Before this fix, it records nothing, and the writer hold stays. ## What Changed - `native-failed-workspace-recovery.ts`: Cancelled-workspace preservation (the delivery worker and the startup scan) now admits a `released` reusable lease with a stopped termination receipt and a stop-and-retain receipt. - `preserveNativeFailedWorkspace` retains such a reusable source only with the exact confirmed stop-and-retain receipt, and only while the workspace is still bound to that writer (`preparedRemoteWorkspaceSource`). Otherwise it writes nothing. - `remote-workspace-writer-hold.ts`: For an ended writer, the holder cutoff now uses the latest progress of an in-progress save of that writer's retained files (a workspace recovery that is not complete, exhausted, or recovered). Reservation and the parked `workspace_busy` wait both read this rule. A waiter waits for the save instead of failing setup. - `doc/execution-semantics.md`: Documents the stopped reusable source and the save-progress cutoff. ## Verification - Red before the fix, green after: - `npx vitest run src/services/native-runtime/native-cancelled-workspace-preservation.test.ts` — new cases "preserves a reassignment-stopped writer's files from its stopped reusable sandbox" (worker and startup scan) failed before the fix: no `workspaceRestoreRecovery` was recorded. - `npx vitest run src/__tests__/heartbeat-remote-workspace-busy.test.ts` — new cases "keeps a waiter parked while an ended writer's save is in progress" failed before the fix: the parked retry was released past the cutoff. - Safety cases: - A reusable source without a confirmed stop, a workspace now owned by another writer, and an already saved workspace are not retained, and the run row does not change. (The writer and saved cases fail if the new guard is removed.) - Preservation does not replay the stopped run, does not create a wake, keeps the stop receipt and lease policy unchanged, and keeps the writer hold until the save publishes. - An exhausted or complete save does not extend the cutoff. The existing past-cutoff test (no save in progress) still fails setup and keeps the held files. - Related suites: workspace recovery, remote workspace reservation, native workspace reconstruction, native failed-workspace recovery, legacy workspace restore recovery. - Not verified on staging: this change is not deployed. The staging task needs this fix deployed. Then the preservation worker records the save, workspace recovery saves the files, and the waiting owner continues. ## Risks - More cancelled runs are now admitted for preservation. The retention guard requires the exact stop-and-retain receipt and the workspace bound to this writer, so a reusable sandbox that another run resumed or that no longer holds unsaved work is not retained. - A stopped reusable source stays `reuse_by_environment` while Paperclip saves it. This matches the existing retention path for reusable native sources. - The writer-hold read adds one indexed query (`issue_recovery_actions_source_run_idx`) for an ended writer only. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell, code editing, test execution) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
7351a9d74f |
fix(native): restore retired backups after warm lease handoffs (#19299)
## Thinking Path > - Paperclip manages AI agents and keeps their work across turns. > - A native session can keep the same sandbox through several warm lease handoffs. > - Each handoff keeps the old lease row and records matching receipts on both owners. > - The retired-backup guard required exactly one row for a provider allocation. > - That rule rejected valid history after Paperclip destroyed the sandbox. > - This change uses the existing handoff proof to select the final owner. > - A verified backup can then restore without discarding ownership checks. ## Linked Issues or Issue Description Refs #19208. **What happened?** A follow-up received a new sandbox after its old sandbox was destroyed. The old allocation had a normal warm lease handoff. Both lease rows remained in PostgreSQL. The guard rejected them with `runner_harness_state_mismatch: backup_without_reusable_lease`. Its later same-run retry failed with `native_remote_recovery_lease_mismatch`. **Expected behavior** Restore the verified backup when one final owner has a valid destroyed receipt and every earlier owner has a matching handoff receipt. Keep uncertain ownership blocked. **Steps to reproduce** 1. Complete a native warm turn. 2. Hand the reusable sandbox to another turn. 3. Complete that turn and save its backup. 4. Close the workspace and destroy the sandbox. 5. Send a follow-up. The new allocation must restore the saved session. **Paperclip version or commit** Reproduced in the PostgreSQL regression on the parent of this change. The affected deployed build also contains #19208. ## What Changed - Select the single final lease owner. Validate all predecessor rows with `retiredReusableLeasePredecessors`. - Keep the destroyed receipt, exact backup stamp, and newer-provider-work checks. - Add database regressions for one and three handoffs. Reject missing or changed receipts, an active predecessor, missing history, an unrelated owner, a stopped source, another backup, and later provider work. - Document the valid lease-history shape. ## Verification - Red before the source change: the two valid-handoff cases returned false. The other 16 database cases passed. - Green after the change: all 18 database cases passed. They also passed on current head `d16e098b8c2e6c0dda70984c1df21e3dc792ab34`. - Command: `pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/native-retired-backup-source.integration.test.ts`. - The database, executor, retry-admission, and lease suites passed: 4 files, 1,308 tests. - `pnpm -r typecheck` and `pnpm build` passed. The build stamp names head `4ee733c3a979436196522f36537ff182e9df5398`. The full local suite stopped in chat shard 3 after a 15-second timeout in `materializes direct external-chat finals once and accepts the next turn without agent bookkeeping writes`. Earlier shards passed 702 cases; that shard passed 303 other cases. The isolated test passes on this branch and unchanged master `10a7d039c8efeac0fd655c24970df5ae3658414b`. All current-head CI chat shards also passed. The earlier full-suite attempt remains failed; its timeout cause is not established. No test or timeout was changed. CI is waiting for its last Runner job. - Current head `d16e098b8c2e6c0dda70984c1df21e3dc792ab34` removes four comment lines from the previously tested head. Its 18 database cases passed. Greptile reviewed this head at 5/5 with no actionable findings: https://github.com/paperclipai/paperclip/pull/19299#issuecomment-6665446098. CI remains pending. - Live inspection found the matching predecessor/successor receipts and the final owner's destroyed receipt. This was read-only evidence from the affected build. - A live provider run at the fixed revision has not been exercised. Local Daytona credentials and an immutable test image are unavailable. No deployment or original-task retry is claimed. - Initial wider tests could not resolve the unbuilt plugin SDK. The SDK build is now available. A sandboxed database launch could not bind loopback; the same regression passed with the disposable database outside that restriction. ## Risks - A wrong owner could allow an old backup to rewind a session. The shared validator must account for every lease row and match both sides of each handoff. - Live, unrelated, incomplete, or conflicting history stays blocked. The existing backup integrity and provider-progress checks remain. - No migration, permission, model, trust, timeout, or retry-policy change is included. - The original task still needs the fixed build and normal recovery. Hosted acceptance remains unverified. ## Model Used OpenAI Codex, based on GPT-6, with reasoning, code execution, and tool use. The session does not expose the exact model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
34d308a331 |
fix(native): let a run retire the prior owner it reserved when its session rotates (#19263)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native Runner agent keeps a warm provider session per task. A follow-up run on the same task can reuse that session and its managed instruction directory. > - When a follow-up run cannot resume the saved session (for example, the effective run configuration changed), it first reserves the prior owner's instruction directory. Then it starts a new session identity and must retire the prior owner. > - Retirement treated the run's own directory reservation as another executor's busy turn. The run failed at startup with `native_session_supervisor_busy`. > - That failure happened after the remote workspace was prepared, so failure handling retained the sandbox and blocked the task. A board follow-up comment left the task `blocked` with no agent progress. > - This pull request lets a run retire the prior owner that it reserved itself, and keeps the directory that it adopted. Any other reservation or turn is still busy. > - The benefit is that a follow-up on a task with a rotated session runs once, and the task does not become blocked. ## Linked Issues or Issue Description **What happened?** A board follow-up comment with reopen woke an agent on a task that had just finished a successful run. The new run reused the task's own sandbox. The saved session could not resume because the effective run configuration changed, so the run started a new session identity. At startup (right after the native execution claim) the run failed with `native_session_supervisor_busy`. The run got the error code `native_continuation_requires_reconciliation`, the sandbox was retained for workspace repair, and the task became `blocked`. The agent had another live run on a different task at the same time. That run used a different sandbox and a different workspace, so it was not the cause. The busy owner was the task's own prior warm session. The new run had reserved that session's instruction directory itself (`reserveWarmNativeInstructionDirectory` sets `preparingRunId` to the new run). `retireSupersededWarmNativeSessions` then saw `preparingRunId` and rejected the run. **Expected behavior** The follow-up run retires the prior warm owner of its own task, keeps the instruction directory it adopted, starts the new session, and runs once. The task does not become blocked. **Steps to reproduce** 1. Run a native Runner (Codex app-server) task to completion with a warm session and a managed instruction directory. 2. Change the agent's effective run configuration, so the saved session cannot resume. 3. Wake the same task again (for example, a board comment with reopen). 4. Before this change, the run fails at startup with `native_session_supervisor_busy` and the task becomes blocked. **Paperclip version or commit** Observed on staging at `7fd05e13`. The code is unchanged on `master`. ## What Changed - `retireSupersededWarmNativeSessions` now receives the current run id. When the prior owner of the same task (same logical owner scope) is reserved by this run (`preparingRunId === runId`), retirement continues instead of throwing `native_session_supervisor_busy`. - That retirement closes the prior owner with `preserveInstructionsForRunId`, so the directory that this run adopted is not collected. This matches the existing configuration-change branch in warm reuse. - After a successful close, the reservation is cleared. A failed close keeps the reservation, so its `release()` retries retirement as before. - A reservation by any other run, a busy owner, and an executing scope still throw `native_session_supervisor_busy`. ## Verification - Red before the fix, green after (`server/src/services/native-runtime/native-session-executor.test.ts`, "managed directory warm checkpoints"): - "retires its own reserved directory owner when the session identity rotates": before, the run rejected with `native_session_supervisor_busy`. After, the prior owner closes once, the new session runs once, the adopted directory is not collected during retirement, and it is collected once when the new owner retires. - "retires its own reserved directory owner after the Runner claim when the session identity rotates" (the production `useRunnerd` path, where retirement happens after the claim): before, admission rejected with `native_session_supervisor_busy`. After, admission retires the prior owner and keeps the adopted directory. - Safety case (green before and after): "still rejects a session identity rotation over another run's directory reservation". The provider does not start and the other run's owner is not closed. - Existing safety case (green): "does not retire an active turn when a new session identity arrives". - `npx vitest run src/services/native-runtime/native-session-executor.test.ts src/services/heartbeat/instruction-working-copy`: 1163 passed. - Staging evidence was read only. This change is not deployed. ## Risks - Low risk. The change applies only when the prior owner has the same logical owner scope (company, agent, task, workspace, environment) and its reservation names the current run. That reservation exists only after this run validated reuse of the prior directory. - The claim-path instruction copy (`claimWarmNativeInstructionCopy`, which sets `busy`) is unchanged. That path is not used by Codex app-server agents. - Two related points are unchanged and are out of scope here: a startup failure after the workspace is prepared still retains the sandbox and blocks the task, and the "Instruction edits could not be recovered" warning on that failure is advisory only. With this fix, this flow no longer reaches that failure. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell, file edits, browser read of staging run records) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Refs #18893 (shared-sandbox ingress retirement). Refs #19143 (workspace-busy wait). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
e37ab23dbc |
fix: hold an agent's new work while its provider usage limit is active (#19246)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Routine-driven agents run on the native Runner through ACP harnesses such as Claude > - When the agent's provider account runs out of allowance, every new wake for that agent fails the same way > - The ACP harness reported the exhausted allowance as a generic "terminal limit failure" with an `unknown` category, so Paperclip could not tell it from a context or turn limit > - Paperclip recorded a reset time only for the failing run's own startup retry; other wakes of the same agent kept dispatching, each spent about two minutes on workspace setup, and then failed > - This pull request classifies the limit, keeps its reset time, and holds the agent's new work until that reset > - The benefit is that a limited agent stops burning runs, shows a clear "waiting for usage limit reset" state with a time, and resumes its queued work after the reset ## Linked Issues or Issue Description Refs #13148 Refs #13379 **What happened?** A routine-driven Claude agent on the native Runner failed 29 times in about 15 hours. 20 runs failed with `native_provider_terminal_failed` and "ACP agent reported a terminal limit failure." The recorded provider failure had category `unknown`. 6 runs failed with `provider_session_open_rejected`. Their `session.open` result is `command_execution_failed` with classification `provider_rate_limit`, and the provider failure is HTTP 429 `rate_limit` ("would exceed your account's rate limit"). The startup retry of one such run recorded a cooldown until one hour later. A new wake for a different task of the same agent dispatched 16 minutes later, spent two minutes on workspace setup, and failed with the same 429. Separately, 4 runs failed with "Project authentication conflicts with this agent's managed AI connection". That message did not say which file or setting conflicted. The same message was also shown when the check could not finish. **What did you expect?** A provider usage limit is classified distinctly. Paperclip records the reset time when the provider reports one. New work for that agent waits for the reset with a visible reason. Queued work resumes after the reset, and no doomed run starts. A configuration conflict is reported only when one is found, and the message names the file to fix. **Steps to reproduce** 1. Use a native Runner agent with a Claude ACP harness whose account is rate limited or out of usage. 2. Let a routine or assignment wake the agent on several tasks. 3. See each run fail with a generic terminal limit failure or a `session.open` rejection, and see new runs keep starting. ## What Changed - Runner: The ACPX turn now receives the typed AIR session failure (`onTerminalSessionFailure`) through the runtime host and the ACPX port. The sidecar adds the provider's title to the provider failure message. Only a rate-limit or quota title becomes `rate_limit` or `quota_exhausted`. Context, turn and configured budget limits stay `unknown`. The failure details stay private. - Server: A native `turn.failed` with a `rate_limit` or `quota_exhausted` provider failure is now `native_provider_usage_limit`, the same code as the Codex usage-limit terminal. - Server: `agentProviderLimitCooldown` (extracted from the startup retry) also reads a reset clock in the provider's wording, such as "resets 2am (UTC)". The clock parser moved from the recovery service to `provider-quota-reset.ts` without changes. - Server: New `heartbeat/provider-limit-wait.ts`. Before any workspace or provider work, a queued run for an agent with an active provider limit is cancelled as `provider_limit_wait`. The run records the reset time, whether the provider reported it, the category and the run that reported the limit. Paperclip schedules one retry of the same wake at the reset through the existing AI connection wait lane. That lane does not consume the failure retry budget. Manual runs, recovering runs, conversation turns and pool-routed connections are not held. - Server: The execution projection shows "Waiting for usage limit reset" with the reset time and the next step. - Server: Only a found project authentication conflict is a configuration failure. Its message names the project file and the settings to remove. A check that cannot finish (timeout or another exit code) is now an ordinary failure, not a configuration conflict. - Docs: `doc/execution-semantics.md` describes the agent-level limit wait. ## Verification Red before the fix, green after: - `server/src/__tests__/heartbeat-provider-limit-wait.test.ts`: An automation wake after a 429 `session.open` rejection dispatched the adapter before the fix. Now the run is cancelled as `provider_limit_wait` and a scheduled retry exists at the reset. After the reset passes, the retry is promoted and the work succeeds. A manual run, a later success, expired evidence and an unclassified terminal still dispatch. - `native-startup-retry.integration.test.ts`: "reads the reset clock from an ACP terminal limit failure's provider wording" (it fell back to one hour before). - `native-session-executor.test.ts`: An ACP `turn.failed` with a typed `quota_exhausted` or `rate_limit` failure is a usage-limit terminal; `unknown` is not. - `packages/paperclip-runner/src/drivers/provider-failure.test.ts`: A typed AIR limit title is kept and classified; context, turn and budget limits are not; details do not leak. - `execution-projection.test.ts`, `ai-connections.test.ts`: The wait label, and conflict versus unfinished check. - Safety cases: `runtime-host.test.ts` and `codex-runtime-adapter.test.ts` forward the callback. The existing startup retry, provider capacity, recovery classification, legacy reconciliation, retry scheduling and execution configuration suites pass. Commands: - `cd server && npx vitest run src/__tests__/heartbeat-provider-limit-wait.test.ts src/__tests__/ai-connections.test.ts src/services/native-runtime/ src/services/recovery/ src/services/execution-projection.test.ts src/services/legacy-execution-recovery.test.ts src/__tests__/heartbeat-retry-scheduling.test.ts src/__tests__/heartbeat-execution-configuration.test.ts` - `cd packages/paperclip-runner && npx vitest run src/drivers/provider-failure.test.ts src/drivers/acpx/runtime-host.test.ts src/drivers/acpx/codex-runtime-adapter.test.ts src/cli/acpx-runtime-sidecar.test.ts` - `tsc --noEmit` for server and paperclip-runner; the ACPX sidecar and provider-failure contract checks pass. I did not run a live provider limit on staging. The live evidence is the staging run records for the affected agent, which I read only. ## Risks - The limit evidence is per agent, not per account. If the board changes the agent's AI connection, automatic wakes can still wait until the old reset (at most one hour when no reset was reported, and at most one day). A manual run is not held, and one successful run clears the evidence. - A new wake that arrives during a limit is now cancelled and retried at the reset, instead of failing. The task stays in its status, and the run shows the reset time. - The Runner change takes effect when the provider pack is rebuilt and deployed. Until then, the server still holds new work using the existing `rate_limit` evidence from `session.open` rejections. - A turn terminal with a typed rate limit or quota is now `native_provider_usage_limit` instead of `native_provider_terminal_failed`. Both are terminal and board-owned. The recovery text now asks the board to restore provider usage capacity. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, browser reads of staging run records) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
5498db1072 |
fix(interactions): relay a verified reply to a self-requested human-only confirmation (#19243)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents ask users to approve proposals and plans with confirmation cards, and users often answer in the conversation > - `resolve-from-comment` lets the responding agent record that verified reply on the card > - Native agents often create their own proposal or plan confirmation as `human_only`, because the `request_human_input` schema offers only that value > - `resolve-from-comment` then refuses with `interaction_human_only`, and the plan and onboarding gates keep the task in `in_review` or `blocked`, although the user explicitly approved > - This pull request adds one narrow relay for that case and keeps every other human-only gate on its card > - The benefit is that an explicit conversational approval works again, and governed, company-capped and quality-review approvals keep their existing controls ## Linked Issues or Issue Description Refs #19073 **What happened?** TestOps launch https://paperclip.testops.cloud/launch/2694 (GitHub run https://github.com/paperclipai/paperclip/actions/runs/41915965118) failed three Product regression cases for the same reason: `continuation/revision-preserves-approval`, `first-task/clarify-propose-accept` and `first-task/revise-accept`. In each case the agent created a `request_confirmation` with `resolverPolicy: human_only` (`effectiveResolverPolicySource: requested`). The user replied "I approve…" or "Yes, I accept that proposal". The agent's `resolve-from-comment` call returned 403 `interaction_human_only`. The task stayed `in_review` (plan gate) or became `blocked` (onboarding decision gate). **Expected behavior** When the agent itself asked for a human-only answer, the verified latest reply of the user it serves is that human answer. The agent can record it. Company-capped, governed-action, issue-review and quality-review gates still require the human to use the card. **Steps to reproduce** 1. A native agent calls `request_human_input` with `interactionKind: confirmation` and `resolverPolicy: human_only`. 2. The conversation user posts an explicit approval comment. 3. The agent calls `POST /api/issues/{id}/interactions/{interactionId}/resolve-from-comment`. Before this change it gets 403. ## What Changed - `issue-thread-interaction-resolution.ts`: the agent actor gets an optional `relayedHumanUserId`. A human-only decision is allowed as `allow_relayed_human` only when all of these are true: - the stored policy is `human_only`, its source is `requested` and its provenance is `explicit` - the resolving agent created the card (`createdByAgentId` matches) - no governed action applies - no additional restriction (issue review) applies - any named addressee user is the relayed user Company caps, the `governed_action` source and legacy rows with an unknown source are still denied. - `confirmation-comment-resolution.ts`: the service sets `relayedHumanUserId` only after its existing comment checks pass: user-authored, not agent-attributed, posted after the card, the latest user message, the same conversation session. It also requires that the author is the card's addressee, the conversation user or the run's responsible user, that the card is not a quality review, and that the company's current cap for the kind is not `human_only` (a tied cap keeps the source `requested`). The same actor goes into the locked accept/reject re-check. The activity entry adds `resolverPolicy: "human_only"` and `resolution: "relayed_human_decision"`. - `routes/issues.ts`: the route pre-check passes the candidate comment author. The service still re-verifies everything under its locks. - `issue-thread-interactions.ts`: carries the relay field into the service's own audience re-check. - Docs: the first-task skill and `api-reference.md` describe the narrow relay. The guidance still says not to add `human_only` to ordinary proposals. ## Verification - `server/src/__tests__/confirmation-comment-resolution.test.ts` (embedded Postgres): 72 pass. - Green: relay through a conversation user and through a run's responsible user. The card is `accepted`, `resolvedByAgentId` and `resolvedByRunId` are set, `resolvedByUserId` stays null, and the audit records `relayed_human_decision`. - Negative controls, each with the card still pending and no audit: agent-authored comment, earlier comment, wrong user, unbound issue/run, `company_cap` source, `governed_action` source, `qualityReview`, `toolAction`, `connectionAuthorization`, issue-review `human_only` restriction, another addressed user, human-created card, another agent's card, inherited company default, and a real capped card created through `svc.create` with a tied cap. The tied-cap test fails as expected when the cap is removed. - Existing negative controls are unchanged, including `human_only` without a bound user and the "rechecks narrowed permissions" retry. - Red check: with the three service files reverted to `master`, the two new green tests fail and the negative controls pass. - `server/src/services/issue-thread-interaction-resolution.test.ts`: 21 pass, including the pure evaluator relay rules. - `server/src/services/native-runtime/confirmation-comment-api.integration.test.ts`: 11 pass. The real route accepts the relayed reply and refuses another user's comment with 403. - Also pass: `issue-thread-interaction-routes`, `onboarding-first-task-admission`, `paperclip-runner-tool-authority`, `runner-api`, `onboarding-first-task-assets` and `company-skills-service` (2,474 tests in total). - Changed files typecheck cleanly. - Live verification: Product regression single-case runs of the three cases are pending after merge. ## Risks - This touches an approval gate. The relay is limited to cards whose creator chose `human_only`. All existing reply-evidence checks still apply, plus a bound-user requirement. Company caps, governed actions, quality reviews, issue-review restrictions and named addressees still deny the relay. - Attribution does not change: the resolver is still the agent and run. The human's comment is kept as `result.commentId` and in the audit. - Legacy rows with a null policy source are not relayed. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
26391707fb |
fix(runner): never settle an OpenCode reply truncated at the output limit (#19242)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native OpenCode runs turn the provider's last assistant text into
the task's visible reply
> - A model can loop until it reaches the output limit. The provider
then reports `finish: "length"`
> - The driver used that cut-off text as the final answer, and the run
succeeded with a broken reply
> - This pull request treats a response that stopped at the output limit
as incomplete
> - The benefit is that users get the accepted completion summary or one
corrected answer, never a truncated wall of text
## Linked Issues or Issue Description
**What happened?**
In a QA staging run, an OpenCode agent on
`openrouter/deepseek/deepseek-v4.1-flash-1017` called `paperclip_finish`
correctly. Then it produced about 32,000 output tokens of repeated text
(`E2E_QUIP_QUIP_...`) until the output limit stopped it. The driver
selected this truncated text as the settled final answer. The run
succeeded, and the task reply was wrong.
**Expected behavior**
A response that the provider stopped at the output limit is not a
complete answer. If an accepted completion report exists, its summary is
the reply. If not, the driver asks for one concise final answer. If that
also stops at the limit, the turn fails.
**Steps to reproduce**
1. Run a native OpenCode turn where the model calls `paperclip_finish`.
2. Let the next assistant message run to the output limit (OpenCode
reports `finish: "length"`).
3. Before this change, that message text is emitted as the final
`agentMessage`.
**Paperclip version or commit**
master at
|
||
|
DottaandPaperclip
|
47027e7ba5 |
fix(routines): retain authenticated agent responsibility on creation (#19222)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Routines save work for later execution under a responsible person's authority. > - An agent can create a routine through its normal REST or native API tools. > - The create route kept the agent creator but dropped the authenticated responsible user. > - The service then selected a parent task's user or a company default, even when the source run had a different user. > - This pull request passes the authenticated responsible user separately from creator attribution. > - Future automatic runs then use the authenticated responsible user saved in the routine revision. ## Linked Issues or Issue Description **What happened?** An authenticated native agent created a routine without a parent task. The stored responsible user became a different company owner. An unrelated parent task could also replace the authenticated user. The defect changes the identity saved for future automatic work. **Expected behavior** Creation must use the server-resolved responsible user from the authenticated run or API key. The agent remains the creator. A person who clicks Run now still supplies that manual run's identity. **Steps to reproduce** 1. Create an active company member and a different company owner. 2. Initialize a native agent run with the member's identity. 3. Use `call_api` to create a paused, self-assigned routine without a parent. 4. Read the routine and its first revision. Before this fix, both use the company owner. **Paperclip version or commit** Reproduced on master `56517dda6c5b32bc901b7dbf1605eaaa041c5fba`. **Deployment mode** Local source integration with real HTTP routes, agent authentication, native tool authority, and embedded PostgreSQL. No provider turn is needed for the reproduction. Related work: [#19128](https://github.com/paperclipai/paperclip/pull/19128) adds board controls for saved responsibility. [#15207](https://github.com/paperclipai/paperclip/pull/15207) repairs synthetic identities from built-in installers. Neither fixes this authenticated create path. ## What Changed - Pass the authenticated agent's responsible user as a separate internal actor field. - Select the board user first, then the agent’s server-resolved responsible user, before the existing parent and company fallback. - Keep creator attribution, initial revision attribution, manual dispatch identity, and existing records intact. - Add real native API, HTTP, and PostgreSQL regressions for identity selection, replay, client spoofing, stale claims, revoked membership, cancellation, cross-company calls, API keys, and fallback behavior. - Document the creation and manual dispatch identity rules in the execution contract. Correct the obsolete responsibility sentence in the agent access map. - Make the existing live-retry route fixture wait for its committed comment’s durable wake before draining. Keep all process and saved-owner assertions. ## Verification - Captured a failing native `call_api` test before changing production code. The persisted responsible user was the unrelated company owner. - The same real route and database boundary passes after the fix. - Initial affected routine and identity checks: 131 passed. The existing pending-AI-card fence also passed. - Follow-up routine checks: 118 passed. The owning scheduled-retry suite passed all 31 tests. - Initial CI serialized shard 7/9 failed with a `23503` wake-to-agent foreign key error during test cleanup. A controlled real-route/PostgreSQL pause reproduced that exact error on unchanged base `56517dda6c`. The drain returned before the route registered its deferred comment wake. The same pause passed after the fixture waited for the actual commentId’s durable wake receipt. The original failure is preserved. - Affected server typecheck after the fixture query change: passed. - `pnpm -r typecheck`: passed. - `pnpm build`: passed. - `pnpm test:run`: still running as supporting coverage. The worktree changed during the run: map cleanup, then the six-line fixture wait. Production and routine-test files stayed unchanged. This is mixed-working-tree coverage, not an immutable full-suite pass for a commit. Exact next-head CI remains required. - `node scripts/check-maps.mjs` and `git diff --check`: passed. - Exact head `a12e3f6c42b8ad5826fd58faa3cd9553ea251e09`: all CI gates passed. Greptile and the native substantive review both gave 5/5 with no open findings. The native reviewer did not run tests or exercise shadow-mode authorization. - Live provider execution remains unverified. This change does not grant access to another person's AI account. ## Risks New agent-created routines now use their authenticated responsible user before a parent or company default. This is the intended behavior change. The service keeps its fallback when no bound identity exists. Missing agent-key identity still fails authentication. Missing personal AI configuration still blocks execution. Existing routines, revisions, tasks, grants, and manual dispatch behavior do not change. The fixture waits for a persisted wake record before draining; its original live-process and saved-owner assertions remain. ## Model Used OpenAI `gpt-6.1-sol` with reasoning, tool use, and code execution. The host did not expose the context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
3a090b815b |
fix: retain native checkpoint ownership through warm close (#19225)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native warm session must save its checkpoint under its exact coordinator owner. > - A failed turn can return an accepted result before that checkpoint is saved. > - The executor stopped renewal and released the owner before the managed session closed. > - The backup owner guard then refused the uncommitted, released owner. A failed close could remain in the warm session map. > - This pull request keeps the claim and its existing renewal through checkpoint, close and instruction collection. > - The benefit is a valid checkpoint owner and one fenced release after the work joins. ## Linked Issues or Issue Description Refs #19208, #19206 and #14695. Those fixes cover a destroyed backup source, Stop before a provider session opens, and retained agent-file ownership. They do not change this checkpoint ownership order. **What happened?** A failed warm turn accepted its result. Its coordinator reached `workspace_finalizing`, with a result ID and no lease owner. Managed close then called the durable checkpoint writer. The writer correctly refused this state with `native_harness_storage_owner_missing`. The failed close could remain for the next fresh session to retire. **Expected behavior** The current claim and renewal remain valid until the required checkpoint, close and instruction collection join. Cleanup stops and joins renewal, then releases only that exact claim. A changed owner is not cleared. An earlier error keeps its identity. **Steps to reproduce** 1. Run the new real-Postgres regression against the base commit. 2. Complete a structured failed warm turn through the normal control-plane result producer. 3. Let the actual managed-session close call its production checkpoint callback. 4. Observe the missing-owner refusal after coordinator release. **Paperclip version or commit** Base: `56517dda6c`. The native ownership order also exists on the earlier failing revision. **Deployment mode** Native warm execution with durable harness storage and a managed remote workspace. ## What Changed - Join warm checkpoint, managed close and instruction collection before coordinator release. - Keep the existing lease renewal active through that work. Stop and join it before release. - Run the same fenced release after a checkpoint or collection error. Preserve the initiating error unless restart detachment has won. Keep the typed restart outcome and changed-owner fences. Attach only closed cleanup-failure flags. - Add real-Postgres ownership, delayed-renewal, error, retry and denial cases. Route the managed-session regression through the existing artifact-equipped native test lane. Update the continuity test for the retained claim. Document the active renewal lifetime. ## Verification - Red: the producing coordinator had an accepted result, `phase=workspace_finalizing` and `leaseOwner=null` at managed close. The production checkpoint callback refused it with `native_harness_storage_owner_missing`. - Green: all 9 real-Postgres regression cases pass. They cover a real 20-second renewal tick, a failed checkpoint, one force-fresh successor, and wrong-boot and changed-owner denial. Actual restart detachment wins even when checkpoint and joined renewal-stop both reject. The replacement claim remains unchanged. No new provider turn or cancellation receipt is created. A private canary does not enter the closed diagnostic flags or an error cause. - The executor, restart-recovery and heartbeat compatibility suites pass: 1,259 tests, with 20 gated skips. The container-enabled continuity case was not run locally. Its 21-second timer check remains. It now requires the exact claim during `hasChanges()` and checks release after the joined work. - All 39 shard-routing and combined-coverage tests pass. The new regression runs in the existing native lane with the required Runner artifact. - Final local head `0edcc37f35` passes `pnpm -r typecheck`, `pnpm build`, `node scripts/check-maps.mjs` and the 9-case regression. The earlier broader compatibility run used identical source bytes. A separate earlier repo-wide local suite on `6cdd68e915` ended with exit 1. Its general-server stage had one GitHub enrollment acknowledgement failure, 31,469 passing tests and 215 skipped tests. That acknowledgement failure reproduced in one narrow run on the same old tree. It is not an established flake or proven caused by this fix. Later repository stages did not run. No duplicate broad suite or full local pass is claimed. - The first published head failed review for detach-error precedence and the old continuity assertion. Its server shard also failed because the new test lacked a Runner daemon in the general lane. Those failures remain recorded. This candidate fixes their causes. Both exact-head reviews are now 5/5. The hosted native-server lane ran all 9 new ownership tests successfully. One unchanged tool-gateway rate-rule test failed in the first current-head CI run. It passed previously and passed locally, both narrowly and in its full 183-test file. The single same-head failed-job rerun passed. All current-head hosted gates are clean. The initial failure remains recorded; no test assertion was weakened. - The provider transport is an in-memory fixture. The managed wrapper, production checkpoint callback, control-plane result and Postgres ownership are real. No paid provider or live remote process is exercised. This does not prove a deployed recovery or a remote stop receipt. ## Risks - A slow required checkpoint extends the lifetime of the existing active renewal. Its cadence stays at 20 seconds for remote claims and 5 minutes for local claims. It runs only while the admitted checkpoint, close and collection remain active, then stops and joins on completion or unwinding. The tests observe a real remote renewal tick. No idle timer, new retry budget or new worker is added. - The release still checks company, task, run, owner, attempt, boot, PID, process start and expiry. Restart detachment retains its recovery owner. - The storage owner guard is unchanged. No migration, permission, provider setting or deployment change is included. ## Model Used OpenAI Codex, `gpt-6.1-sol`, with reasoning, source inspection, local code execution and read-only staging metadata. Context window and reasoning effort are host-managed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
beb73381ca |
fix(native): let a Stop before the provider session opens continue once (#19206)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A remote task restores its saved workspace into a sandbox before the agent starts. > - When that restore fails, the run records only `remote_workspace_operation_failed: operation=restore exit=1`. > - The run then fails as `setup_failed`. No retry starts, and a `blocked` task gets no recovery action. > - An operator cannot tell which step failed, and nobody owns the next step. > - This pull request makes the sandbox scripts name their failed step in a closed grammar, and it gives a restore failure a bounded retry or a named board recovery action. > - The benefit is a visible cause and a clear owner. Saved work stays protected. ## Linked Issues or Issue Description Refs #19152 and #19187. Those changes name copy-back (save) failures. Refs #18852 and #19195. Those changes added the bounded bootstrap retry that this change reuses. **What happened?** A board reply woke an agent on a `blocked` parent task. The run failed during setup 13 seconds later. The only run event was `remote_workspace_operation_failed: operation=restore exit=1`. The run had no workspace operation row. The task stayed `blocked`, and its recovery actions showed `active: null`. The same opaque family also appeared for workspace preparation (`exit=1` only). **Expected behavior** The run names which restore step failed, without command output or paths. A failure before provider work uses a bounded retry when a retry can help. Otherwise, and when the retry budget is spent, the board gets a named recovery action. The saved version is never changed or overwritten. **Steps to reproduce** 1. Save a remote workspace, then start a new run on a different allocation. 2. Put a folder at the published workspace path in the sandbox before restore. 3. Observe that the base code records only `operation=restore exit=1`, then `setup_failed`, with no recovery action for a `blocked` task. **Paperclip version or commit** Base: `87dfba6818`. **Deployment mode** Legacy runtime with sandbox (Daytona) remote workspace storage. ## What Changed - The prepare, repository, and restore sandbox scripts track their current stage. On failure they print `remote_workspace_<op>_failed: stage=<stage>` plus closed fields: `reason=` (the script's own snake_case refusal), `archive=` (the archive helper's fixed reason), `code=` (errno), `disk=full`, `git=` (a fixed label chosen from Git output), and the existing `exit`/`signal`/`program` fields. - The server accepts that text only when it matches a strict grammar for the same operation. Other output is dropped, as before. - Provider refusals during the restore upload and restore command are recorded by closed labels, like copy-back. - A failed saved-workspace restore raises `RemoteWorkspaceRestoreError` with the server step (`snapshot_lookup`, `snapshot_download`, `sandbox_restore`). It is retryable only when the owner re-reads the same writer, epoch, and saved version with no unsaved work, and the sandbox did not name a refusal reason. - Setup failure handling classifies a pre-dispatch legacy restore failure as `remote_workspace_restore_failed`. A retryable failure uses the existing bounded bootstrap retry. A non-retryable or exhausted failure records a board-owned recovery action with the redacted cause, when stranded-work recovery does not scan the task status (for example `blocked`). - Stranded-work recovery gets a named notice for this code. Generic continuation recovery does not grant it a second budget, and legacy reconciliation does not invent provider actions for it. - `doc/execution-semantics.md` documents the contract. ## Verification - Red: the new transport tests failed on the base code with `remote_workspace_operation_failed: operation=restore exit=1` (and the same for prepare and repository). The new completion tests failed because the run became `setup_failed` with no retry and no recovery action. - Green: `npx vitest run src/services/remote-workspace-transport.test.ts src/services/heartbeat/run-completion.test.ts src/services/remote-workspaces.integration.test.ts src/__tests__/heartbeat-retry-scheduling.test.ts src/services/legacy-execution-recovery.test.ts src/services/recovery src/services/remote-workspace-save-concurrency.test.ts src/services/native-runtime/native-workspace-finalizer-failure.test.ts src/__tests__/native-workspace-finalization-work.test.ts` passes (563 tests). - The transport tests use real sandbox commands. They prove an occupied folder reports `stage=inspect_root reason=occupied_root` and keeps its files. They prove a wrong manifest reports `stage=extract archive=manifest_mismatch`. They prove a missing branch reports `stage=checkout_branch`, and a changed origin reports `stage=verify_origin`. Forged output with a path is dropped. - The real-PostgreSQL cold-restore test proves an occupied root gives a non-retryable `sandbox_restore` failure and keeps the unknown files. - The completion tests cover retried, exhausted, refused, post-dispatch, and `in_progress` cases. - `tsc --noEmit` for the server reports no errors in the changed files. ## Risks - A retry keeps the same restore path. If the cause is deterministic but unnamed, the run uses its two retries before the board action. Named refusals skip the retries. - The board action is recorded only for statuses that stranded-work recovery does not scan, and only when no other action is active. This prevents a second recovery identity for one failure. - Script stderr grammar is new. Old sandboxes run the script that the server sends, so no runner update is necessary. - No migration, deployment, or credential change. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use, code execution, and browser tools for read-only staging evidence. Context window and reasoning mode are host-managed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
f5891564ec |
fix(recovery): continue from a retired sandbox's backup and hand spent export repairs to the board (#19208)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native tasks keep a warm sandbox and a verified per-turn backup of the provider session. > - Recovery holds stop a task when continuation is not safe. They must have a next action and an owner. > - In a staging soak, two kinds of held tasks could not move forward. Users followed the product's own instruction, and nothing changed. > - First, a reply to a task whose warm sandbox Paperclip had already destroyed failed every time. The label then said "Send a message to continue" again. > - Second, an exhausted system workspace repair stayed active and system-owned for hours. The board saw no owner, and a reply started nothing. > - This pull request lets a reply continue from the backup when the source sandbox is provably gone, and hands a spent system repair to the board with a named Retry. > - The benefit is that productive work always has a next action that can succeed, and nothing replays or runs twice. ## Linked Issues or Issue Description **What happened?** (a) A native warm task ran one successful turn. A later legacy follow-up retired the warm sandbox lease. The next user reply got a new sandbox (`created` lease). The executor found a verified backup but no reusable lease, so it threw `runner_harness_state_mismatch: backup_without_reusable_lease`. The run retried on the restart-recovery path, failed `native_remote_recovery_lease_mismatch`, and ended as `native_continuation_requires_reconciliation`. Workspace recovery then said "Saved work recovered. Send a message to continue." Each new reply repeated the same failure. (b) A run failed `native_workspace_sync_out_retry_exhausted`. System workspace recovery used its 6 bounded attempts and became `exhausted`. The recovery action stayed `active` with `ownerType: system`. The UI hides system-owned recovery actions, so the board saw no action. Nothing ran for more than two hours. **Expected behavior** (a) When Paperclip destroyed the source sandbox and no provider work followed the backup, the reply continues once from the backup. Ambiguous states still fail closed. (b) When the bounded budget is spent, the action names a board owner and a Retry action that can progress it. **Steps to reproduce** (a) Finish a warm native turn. Retire its reusable lease through obsolete-lease cleanup. Send a reply. The new run gets a `created` lease and fails `backup_without_reusable_lease`. (b) Let system workspace recovery reach its deadline or retry limit. Read the recovery action. It is still `ownerType: system`. Related merged work checked first: #19145, #19184, #19187, #19189, #19190, #18943 (added the guard for migrated sessions). Open recovery PRs (`gh pr list --search recovery --state open`) do not touch these paths. ## What Changed - `native-retired-backup-source.ts` (new): `retiredSourceAuthorizesNativeBackupRestore` decides whether a created lease may restore the backup. All must be true: the backup's source lease is `expired` or `failed`; it has a provider termination receipt with state `destroyed`; its `nativeHarnessBackup` stamp names this exact manifest SHA-256, session and runner instance; exactly one host lease maps to the provider allocation; and no provider durable event for the session was recorded after the backup completed. The provider event type set moved here and the executor imports it. - `native-session-executor.ts`: `shouldRestoreNativeHarnessBackupIntoSandbox` accepts `retiredSourceContinuation`. A created lease with a configured reusable lease evaluates it, beside the existing migrated-session path. The restore uses the existing verified restore and stamps the new lease. - `system-recovery.ts`: `updateSystemRecovery` moves an `exhausted` action to `ownerType: board` and writes a next action and workspace recovery message that name the attempt count, the last failure code, and **Retry recovery**. Retry (`requestWorkspaceRecovery`) already starts a new bounded generation and returns ownership to the system. - `doc/execution-semantics.md`: documents both rules. ## Verification - Red before the fix, green after: - `npx vitest run src/services/native-runtime/native-session-executor.test.ts -t "restores a warm backup into a created lease only when"`. Before: the destroyed-source case rejected with `runner_harness_state_mismatch: backup_without_reusable_lease`. After: it restores the runner state and provider home into the new sandbox and stamps the new lease. - `npx vitest run src/services/workspace-recovery.test.ts -t "hands an exhausted system export repair"`. Before: `exhausted` with `ownerType: "system"`. After: `ownerType: "board"`, the blocker says Retry recovery and never says "send a message", no provider run or lease resume happens, and Retry starts one new system generation. - Safety cases (fail closed): a stopped (not destroyed) source, a stamp for another checkpoint, provider work after the backup, a still-reusable `released` lease, a missing receipt, and two host leases for one allocation. These run in the executor test and in `native-retired-backup-source.integration.test.ts` against embedded Postgres. - Related suites pass: `workspace-recovery.test.ts`, `system-recovery-state.test.ts`, `native-session-executor.test.ts`, `task-retry.test.ts`, `explicit-native-continuation.test.ts`, `workspace-recovery-source.test.ts`, `native-workspace-export-retry.test.ts`, `native-remote-retry-admission.test.ts`, `native-workspace-warm-reuse.test.ts`, `native-workspace-sync-retention.test.ts`, `heartbeat-observed-workspace-preservation.test.ts`, `execution-control-work.test.ts` (2,224 tests). - `npx tsc --noEmit -p server/tsconfig.json` passes. - Not done: a live run against staging. Staging access was read-only. The staging evidence is the run event timeline described above. ## Risks - (a) A backup restore after a lost sandbox could rewind a session if newer provider state existed. The predicate requires a provider `destroyed` receipt and no provider event after the checkpoint. A stopped sandbox, a pending cleanup, or a still-reusable lease still fails closed with the old error. - (a) The event-time check can fail closed if a late event from the backup turn is written after the checkpoint time. That keeps the old behavior; it never authorizes a restore. - (b) The action's next action text stays as the exhausted message while a later Retry generation runs. The task blocker shows the live workspace recovery message first, so users see "Recovering saved work…" during that time. - Not changed: when the predicate fails closed, the run still follows the existing reconciliation path. That path can still end with the "Send a message to continue" receipt. That residual is narrower now and needs a separate change to the workspace recovery receipt. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic. Agentic coding with tool use (shell, file edits, read-only browser inspection of staging). Default context window. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
87dfba6818 |
fix(native): retry an unpublished accepted task reply durably instead of dropping it (#19200)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run commits its status decision and its final reply comment
in separate steps
> - Background presentation recovery publishes an accepted reply when
the live path does not record one
> - That worker dropped the run after one refused attempt and only threw
on an error, with no record on the run
> - A task could then show done while the user never saw the answer,
until an unrelated wake reloaded the queue
> - This pull request makes the publication of an accepted ordinary task
reply durable, bounded, and recorded
> - The benefit is that a task cannot close silently without its answer
## Linked Issues or Issue Description
Refs #19130 (verify requested publication at finalization, for
documents), #19183 (deliver messages saved while a native run
finalizes), #19186 (stale owner done). This change does not duplicate
them. It covers the final reply comment.
**What happened?**
A native run accepted a result with disposition `done`. The status
decision set the task to done. The accepted final reply was not
published as a task comment. The run had no publication record (no
`run.presentation.resolved` event and no `presentationDecision`). The
reply was published by background recovery 58 minutes later, after an
unrelated wake reloaded the presentation queue.
**Expected behavior**
The accepted reply is published promptly. A refused or failed attempt is
retried with backoff and never posts twice. If publication fails
permanently, the run and the task show a named failure with an owner.
**Steps to reproduce**
1. Commit an accepted native task result with disposition `done`, with
the final reply not yet visible to presentation recovery.
2. Run the native response presentation worker check once. The attempt
is refused.
3. On `master`, the worker returns `null` and drops the run. Nothing
records the attempt. Nothing retries it until another notification for
that scope.
**Paperclip version or commit**
`master` at
|
||
|
Devin FoleyandPaperclip
|
07a2941f0c |
fix(recovery): reclaim verified inactive provider caches on full disks (#19192)
Recover from a full-disk workspace save by reclaiming only a verified inactive provider installation under the existing source and maintenance authority. Commit the preparation inventory in Core before changing cache contents. Bind interrupted cleanup to stable filesystem identities and keep small staged generation markers when directory birth times are unavailable. Skip read-only or permission-denied assets and stop freeing blocks once marker space is available. Preserve task files, Git, session history and uncertain action receipts. Only the normal saved-workspace and provider-stop receipts permit continuation. Validated with 446 complete recovery regressions on the repair revision, subsequent focused recovery and transport suites, typecheck/build, and all exact-head hosted CI. Greptile reviewed the final head at 5/5 with no unresolved findings. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
245b4d3b79 |
Preserve cancelled workspaces after controllerless Stop (#18820)
Admit cancelled workspace preservation after an audited Stop acknowledgement when no live controller received an interrupt. Retain the strict dispatched receipt for provider termination and operator handoff. Existing source, lease, coordinator and result ownership checks remain enforced. Validated with complete Stop/preservation suites (134 tests), real-file recovery, workspace typecheck and build, and green hosted CI. Greptile and Paperclip both reviewed the exact head at 5/5 with no unresolved findings. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
3b17b5ce15 |
fix(native): keep a reassigned-away run from closing the new owner's task (#19186)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native agent run reports "done" in an accepted result. A finalizer then decides the task status. > - The board can reassign a task while the run is in flight. The old run is cancelled, but its accepted result still goes to the finalizer. > - The finalizer did not check who owns the task now. The old owner's "done" closed the task of the new owner. > - The new owner never did the work, and the task shows done with no reply from the current owner. > - This pull request makes a done report apply only when the reporting run's agent still owns the task. > - The benefit is that reassignment keeps its meaning: the new owner keeps the open task, and the old result stays as history. ## Linked Issues or Issue Description **What happened?** On a hosted staging instance, agent A started a native run on a task. Agent A finished the work and submitted a "done" result. Then the board reassigned the task to agent B, and Paperclip cancelled agent A's run with `issue_reassigned`. About 45 seconds later, the native finalizer committed agent A's done decision with reason `completion_claim_policy_accepted`. The task became done. Agent B's runs were then cancelled because the task was terminal. Agent B never worked on the task. **Expected behavior** A done report from a run that no longer owns the task must not close the task. The accepted result stays as history. The task status stays as it is for the current owner. **Steps to reproduce** 1. Assign a low-risk native task to agent A and start a run. 2. Submit a done result for the run. 3. Reassign the task to agent B before finalization. 4. Finalize the run. Before this change, the task becomes done. **Paperclip version or commit** Observed on staging at `b9fbbf8d30`. Reproduced on `master` at `06826915d7`. Related: #19130 adds a publication check for requested documents at finalization. It does not check task ownership. ## What Changed - `status-arbiter.ts`: add `ownsNativeTaskCompletion` and `supersedeStaleOwnerCompletion`. A done decision from an agent that is not the current assignee (another agent, a user, or nobody) becomes `preserve` with reason `completion_owner_superseded` and no effects. Other decisions do not change. This is the same ownership rule that the provider-failure path already uses (`ownsProviderFailureDecision`). - `native-run-finalizer.ts`: apply the rule after arbitration and before the deliverable check. Conversation tasks keep their existing path. - `native-status-reassessment.ts`: apply the same rule, so new evidence from a reassigned-away run cannot close the task later. - The status committer does not change. A reassignment during finalization already increments `statusVersion` (migration 0358). The committer then replans, and the retry applies the new rule. - `doc/execution-semantics.md`: describe the rule next to the existing "a late finalizer cannot change a reassigned task" statement. ## Verification New file `server/src/services/native-runtime/native-stale-owner-completion.test.ts`. It uses embedded PostgreSQL, the real coordinator store and the real finalizer. - Control: the owning run's done result closes the task with `completion_claim_policy_accepted`. - Reassigned before finalization: the task stays `in_progress` with agent B. The decision is `completion_owner_superseded`. The accepted result row is unchanged. - Reassigned after arbitration and before the status commit: the commit replans and the task stays `in_progress`. - Reassessment control: later approved evidence closes the owner's task with `decision_superseded_by_new_evidence`. - Reassessment after reassignment: later approved evidence from the old run does not close the task. The decision is `completion_owner_superseded`. This case fails when the reassessment rule is disabled. - Pure policy cases: another agent, a board user and no assignee all preserve status. A non-done decision passes through. Red before the fix (same tests, fix not applied): ```sh cd server && npx vitest run src/services/native-runtime/native-stale-owner-completion.test.ts # Tests 2 failed | 1 passed — both reassignment cases: expected "in_progress", received "done" ``` Green after the fix: ```sh cd server && npx vitest run src/services/native-runtime/native-stale-owner-completion.test.ts # Tests 10 passed cd server && npx vitest run src/services/native-runtime/paperclip-control-plane-port.test.ts \ src/services/native-runtime/native-completion-feedback.test.ts src/services/native-runtime/native-deliverable-feedback.test.ts \ src/services/native-runtime/native-skill-test-status.test.ts src/services/native-runtime/native-routine-status.test.ts \ src/__tests__/native-status-arbiter-corpus.test.ts src/__tests__/native-status-reassessment-work.test.ts \ src/services/native-runtime/agent-chat-response-recovery.test.ts src/__tests__/native-run-projection-work.test.ts \ src/services/native-runtime/task-reassignment.test.ts src/services/native-runtime/native-provider-failure.integration.test.ts # 11 files, 614 tests passed ``` I did not run the full `pnpm test:run`, `pnpm -r typecheck` or `pnpm build` locally. CI runs them. ## Risks - A native run that reassigns its own task to a user or to another agent, and then reports done, no longer closes the task through arbitration. The status that the agent set through the API stays. This matches the existing provider-failure ownership rule. - A native run on a task with no assignee cannot close it through arbitration. Native task runs start from an assignment, so this is not expected in normal use. - Review runs and conversation tasks keep their existing paths. Review outcomes already preserve status before this rule. - No schema change and no migration. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Provider: Anthropic. Model: Claude Opus 5.5 (`claude-opus-5-5`), in the Claude Code agent harness. - Capabilities used: tool use, shell and code execution, repository search, browser reads of staging, and extended reasoning. - The model wrote the change, the tests and this description. A human requested the fix. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
2141efc097 |
fix(native): answer Stop during slow native startup with 409 and settle it as cancelled (#19189)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Operators stop a running agent run with `POST
/api/heartbeat-runs/:runId/cancel`
> - For a native run, Stop waits up to 30 seconds for the native session
to open before it can deliver the cancel
> - A remote native session can take longer than 30 seconds to open.
Then the wait fails with `native_cancellation_pending_recovery`, and the
route answers HTTP 500
> - The Stop intent was already saved. The startup owner fenced the late
session in memory only, so the run ended as a provider failure, not as a
cancellation
> - This pull request maps that outcome to a clear 409 and makes the
late session acknowledge the saved Stop intent
> - The benefit is a consistent cancel result and a run that ends as
`cancelled` when an operator stops it during a slow startup
## Linked Issues or Issue Description
**What happened?**
An operator stopped a native run about 40 seconds after it started. The
run was still opening its remote provider session (session startup took
about 105 seconds). The cancel request waited 30 seconds for the startup
handle, then returned HTTP 500 `{"error":"Internal server error"}`. The
run record showed `executionCancellation.state = "requested"` and
`nativeCancellation.dispatchState = "pending"`. When the session opened,
the executor fenced it and threw `native_finalization_missing: session
returned no semantic result`. The run ended `failed` with
`native_session_interrupted`, and the pending Stop intent was never
acknowledged.
**Expected behavior**
The cancel route returns a clear result and never an internal error. If
the Stop is saved but the session is still starting, the route returns
409 with code `native_stop_pending`. When the late session opens, it
acknowledges the saved Stop, and the run settles as an operator
cancellation.
**Steps to reproduce**
1. Start a native run whose provider session takes more than 30 seconds
to open.
2. Stop the run from the board while the session is still opening.
3. See HTTP 500 from the cancel route, and see the run end `failed`
instead of `cancelled`.
## What Changed
- `server/src/services/heartbeat/run-control.ts`: the cancel path
catches `native_cancellation_pending_recovery` from native dispatch. It
throws `409 native_stop_pending` instead. The run stays `running` and
keeps its saved Stop intent. No terminal transition or release happens.
- `server/src/services/native-runtime/native-session-executor.ts`: when
a late startup handle opens after a Stop caller timed out, the executor
reads the run. If a run-scope Stop intent is still `pending`, it calls
the durable `cancelNativeSession(..., { db, scope: "run" })` against the
published handle. This records the audited acknowledgement (`dispatched:
true`). The existing in-memory fence still runs. If the acknowledgement
fails, the executor logs a warning and keeps the previous behavior.
- New regression
`server/src/services/heartbeat/native-stop-pending.test.ts` (real
Postgres): startup-wait timeout and remote-ack timeout both give 409
`native_stop_pending`. The run stays `running` with its Stop intent.
Other native Stop errors stay unchanged.
- New regression in `native-session-executor.test.ts`: a Stop that times
out on startup gets its saved intent acknowledged when the late handle
opens. The provider session is cancelled once, and no turn is submitted.
## Verification
- Red before the fix: `native-stop-pending.test.ts` failed (raw
`NativeCancellationPendingRecoveryError` / `Error`, not `HttpError`).
The executor regression failed (no acknowledgement write).
- Green after the fix: `cd server && npx vitest run
src/services/heartbeat/native-stop-pending.test.ts`
- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t "late
startup"`
- Related suites:
`src/services/native-runtime/native-session-executor.test.ts`,
`src/services/heartbeat/`,
`src/services/native-runtime/native-stop-accounting.integration.test.ts`,
`src/__tests__/agent-live-run-routes.test.ts`.
- Evidence: the incident run had cancellation recorded at 01:12:01Z and
the 500 at 01:12:32Z. This matches the 30-second
`waitForNativeSessionStartup` deadline. Its final error matched the
startup-stop branch in the executor.
## Risks
- Low to medium. The 409 only replaces an error that was already a
failure. The caller still sees "not stopped yet", and the run is not
marked cancelled.
- The late acknowledgement runs only when a run-scope intent is
`pending` and the handle is published. It uses the same durable cancel
path that the reattach branch already uses. It can commit the normal
Stop status decision for the issue. That is the same effect a timely
Stop has.
- If the acknowledgement fails, the code falls back to the previous
in-memory fence. A Stop never gives a second executor: no provider turn
is submitted after the handle opens.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandClaude Opus 5.5
|
f1516e8009 |
fix(native): stop a session.open model rejection instead of a doomed bootstrap retry (#19190)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native Runner runs start a provider session with the `session.open` command before any prompt > - A definite `session.open` rejection is treated as a pre-session failure and gets a bounded same-run bootstrap retry > - The runner also reports a closed classification. Two classifications say that the provider cannot run the selected model: `requested_model_unsupported` and `grok_model_mismatch` > - Paperclip ignored that classification. It scheduled a retry that must fail the same way. On a remote sandbox with a retained runner PID, the due retry entered restart recovery. Restart recovery has no checkpoint for a bootstrap retry, so it failed closed as `native_restart_recovery_blocked` / `ambiguous_remote_recovery_evidence` > - The task then stayed `blocked`. The recovery actions settled as no-replay holds, so no active action told anyone to change the model, and the parent task stalled too > - This pull request classifies those two definite rejections as the existing `native_provider_model_rejected` failure > - The benefit is a terminal failure on the first attempt, an accurate error, and an active board recovery action that names the next step: choose a supported model ## Linked Issues or Issue Description **What happened?** On a hosted staging instance, a Grok agent run failed at `session.open`. The runner log said `ACPX sidecar command session.open was rejected (retryable=false, classification=grok_model_mismatch)`. Paperclip recorded `recoveryMode: "bootstrap_retry"` and scheduled a same-run retry 30 seconds later. The run had a remote sandbox and a recorded sandbox PID, so the due retry went to restart recovery. Restart recovery found no checkpoint and no provider events and failed closed with `ambiguous_remote_recovery_evidence`. The run ended as `native_restart_recovery_blocked`. The task stayed `blocked` with no active recovery action for more than one hour. Its parent task stayed blocked on it. **What did you expect to happen?** The selected model cannot start, so a retry cannot succeed. The run should fail once with the model-rejection error. The task should show the existing "choose a model" action, owned by the board, until someone changes the model and retries. **Steps to reproduce** 1. Start a native run whose `session.open` command fails with `classification=grok_model_mismatch` (or `requested_model_unsupported`). 2. Read the `native_run_finalizations` row. Before this change it is `retryable_failure` with `recoveryMode: "bootstrap_retry"` and a `nextAttemptAt`. 3. On a remote sandbox run with a recorded PID, the due retry enters restart recovery and fails as `native_restart_recovery_blocked`. **Paperclip version or commit** `master` at `f5f5184f23`. Related: #18910 routes due remote retries with a retained PID to restart recovery. #18908 continues verified failed native preparation. Open PRs #19177, #19180, #19182 and #19184 change no-replay hold settlement and workspace hold release. They do not change this classification. This PR does not duplicate them. ## What Changed - `nativeSessionFailureSourceCode` (`server/src/services/native-runtime/native-session-executor.ts`) returns `native_provider_model_rejected` for a definite `session.open` failure or rejection whose runner classification is `requested_model_unsupported` or `grok_model_mismatch`. Other definite `session.open` rejections still return `provider_session_open_rejected` and keep the bounded startup retry. An indeterminate `session.open` outcome stays `native_session_interrupted`. - `native_provider_model_rejected` is already a permanent failure. The run stops on the first attempt with no `nextAttemptAt`, so it never enters the restart-recovery retry path. - When no provider session exists, the coordinator `nextAction` and the board recovery action `nextAction` say: choose a supported model for this agent, then retry the task. - The existing product paths for `native_provider_model_rejected` apply: the task presentation shows `choose_model`, and the recovery service treats it as `configuration_incomplete`. - `doc/execution-semantics.md` describes the rule under "Native provider model capacity". ## Verification Red before the fix (new test, unchanged production code): ```sh cd server && npx vitest run src/services/native-runtime/native-startup-retry.integration.test.ts # Tests 2 failed | 21 passed # failureCode: native_session_interrupted, phase: retryable_failure, recoveryMode: bootstrap_retry, # nextAction "Retry provider bootstrap on this same run ..." - the same record seen on staging ``` Green after the fix: ```sh cd server && npx vitest run src/services/native-runtime/native-startup-retry.integration.test.ts # Tests 23 passed cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts \ src/services/native-runtime/native-startup-retry.integration.test.ts src/services/task-interruption-presentation.test.ts # Test Files 3 passed, Tests 1186 passed pnpm --filter @paperclipai/server typecheck # exit 0 ``` The new integration cases use embedded PostgreSQL and the real executor failure path. For both classifications they check: - coordinator `terminal_failure`, `failureCode: native_provider_model_rejected`, `nextAttemptAt: null`, attempt 1 - run `failed` with `errorCode: native_provider_model_rejected` - task `blocked`, execution lock released - an `active` board recovery action with cause `native_provider_model_rejected`, return owner set to the agent, and the choose-model next action - `settleUnrecoverableExecutions` leaves the action `active`, so it does not become a silent no-replay hold - `claimNativeSessionResumptions` claims nothing, so there is no second executor The unit test also checks that `session_ensure_failed` keeps its bounded retry and that an indeterminate model rejection stays unclassified. A live Grok run with a mismatched model was not run. The failure record in the test matches the staging record field by field. ## Risks - Low risk. The change adds two closed runner classifications to an existing permanent failure code. Other `session.open` rejections keep their current retry. - A model mismatch that a retry could fix would now need an explicit retry. The runner already marks these rejections `retryable=false`. - This PR does not change restart recovery. A remote bootstrap retry with a retained sandbox PID for other `session.open` causes can still fail closed as `ambiguous_remote_recovery_evidence`. That needs a separate change with remote stop evidence. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used in Claude Code with tool use (shell, file edits, browser reads of the staging API). It did the investigation, code change, tests and PR text. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
f7a4c5678f |
feat(runtime): pre-copy completed harness history before enrollment (#19137)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - An ephemeral web server needs provider history in shared object
storage.
> - Offline enrollment must preserve old backups before it can detach a
volume.
> - A large copied dataset still exceeded the maintenance deadline with
parallel transfers.
> - Most of its bytes belong to successful turns with released runtime
authority.
> - This pull request pre-copies that history while unrelated executions
continue.
> - The final offline migration still verifies every source and grants
enrollment authority.
## Linked Issues or Issue Description
Refs #19108, #19038, #18943.
**What happened?**
A copied instance has 1,953 local harness backups and 54.8 GB of
declared source bytes. A resumed parallel migration still reached its
25-minute deadline. It cannot safely enroll by moving all history during
maintenance.
**Expected behavior**
Copy eligible completed history while the server remains online. Defer
active, failed or changed sources. Keep all existing data and final
enrollment checks.
**Steps to reproduce**
1. Retain many local native harness backups on a volume-backed instance.
2. Attempt a full offline conversion within the maintenance deadline.
3. Observe that most time moves historical bytes rather than current
execution state.
## What Changed
- Add finite `native-harness-prestage.mjs` preparation from a normal
runtime build. Require explicit opt-in and shared object storage.
Process one source at a time with a 90-minute deadline.
- Use the recorded released runtime owner of each successful, finished
turn. Do not assert that all writers stopped.
- Recheck checkout, related queued/running work, successful result
ownership, session, Runner, exact lease stamp and unchanged backup at
database publication fences.
- Defer source changes, malformed source JSON and unsafe backup roots.
Log each such source and its reason. Keep database failures and
cancellation fatal. Keep completed objects valid for retry. Never return
an enrollment-ready receipt.
- Reuse the bounded migration source walker. Keep offline admission and
final verification unchanged.
- Keep private lifecycle state, task ownership and provider policies
unchanged. Document the operator path and its limits.
## Verification
- A zero-source scan now checks the existing database obligation query.
A known mismatch blocks with a source-missing receipt; empty
fresh/already-durable scans and nonempty deferred scans remain valid
advisory results. The prior command fails the mismatch regression. All
16 command and runtime-bundle tests pass for this follow-up.
- At head `7ba3dbc7aa`, 87 focused command, database-timeout, storage,
archive, publication and runtime-bundle tests passed with real
PostgreSQL and filesystem restoration.
- The current follow-up passes all 68 backup and archive tests. A source
renamed after preparation fails the new regression on the previous head,
then defers and permits the next backup with this fix. An archive output
ENOENT remains fatal. Existing database-error and cancellation tests
pass.
- New race tests fail against the previous storage owner: a changed
backup, queued turn or revoked lease otherwise publishes.
- Tests prove unrelated executions stay running, same-task work defers,
failed history stays deferred, retry verifies existing objects, and
private bytes restore after deleting the old home.
- Command review fixes require the configured existing vault key, retain
it for the process, set database lock/statement/idle bounds, close
clients on cancellation, and cap shutdown at ten seconds. Abort
immediately sets failure status and prints one blocked receipt. A real
child with real shortened timers reproduces the old missing-receipt exit
and verifies the fix. Tests also use a real PostgreSQL lock.
- Malformed manifest or runner JSON and symlink replacement fail the new
preparation and publication-fence regressions on the previous code. The
fixed command defers each damaged source and copies the next eligible
backup. Separate tests keep database SyntaxError and cancellation fatal.
Offline migration still rejects damaged sources.
- The bundle test runs the emitted preparation entry with plain Node
without starting the app.
- A prior copied-data experiment completed with 1,738 new publications,
nine verified current objects and 206 deferred sources. Later checks
found different database and filesystem capture times: 43 source runs
were absent from the older database snapshot, and one required
historical manifest was absent from the retained filesystem archive.
This experiment does not prove a consistent restore or enrollment
readiness.
- Full current-head CI at
|
||
|
DottaandClaude Opus 5.5
|
a994a648f1 |
fix(workspaces): name copy-back failures on the run and end failed save operations (#19187)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native run in a Daytona sandbox saves its workspace after the provider turn. This save is "copy-back". > - On staging, about 20 runs kept a saved result, but copy-back failed three times. The issues became blocked with `native_workspace_sync_out_retry_exhausted`. > - #19152 added a redacted cause for a sandbox command that exits with an error. Three gaps stayed open: a provider that refuses the command still gives a bare code, the run does not show the cause, and the failed save operation stays "running". > - The first failure also had the wrong name. The runner kept the result but did not prove a durable suspension. The executor reports this as `native_session_lease_lost`, and the workspace operation shows only that text. > - This pull request names each of these failures with closed, redacted labels. It also ends a failed save operation as "failed". > - The benefit is that operators can see why copy-back failed and which save is still active. Retries and saved results do not change. ## Linked Issues or Issue Description Refs #19152, #19145. These are the earlier copy-back diagnostics and retry-scheduling repairs. This PR does not duplicate them. **What happened?** A native task ran in a Daytona sandbox and its result was accepted. The run then failed with `native_workspace_sync_out_retry_exhausted`. The run had no error text. The automatic copy-back retries each failed in under one second, with only `workspace_sync_out_failed`. The final `remote_workspace_snapshot` operation stayed "running" after the retries stopped. The first `workspace_finalize` operation showed only `native_session_lease_lost`. The run log showed that the runner shutdown timed out before durable suspension. The lease was not lost. **Steps to reproduce** 1. Run a native task in a Daytona sandbox until its result is accepted. 2. Make the sandbox provider refuse the copy-back command (for example, a lease that is no longer active), so that no exit status comes back. 3. Let the automatic retries stop. Then read the run and its workspace operations. **Expected behavior** The run, the coordinator, and the failed operations name a redacted cause. The failed save operation is "failed" with a finish time. **Actual behavior** The run has no cause. The save operation stays "running". A provider refusal keeps only the bare `workspace_sync_out_failed` code. ## What Changed - `remote-workspace-transport.ts`: the provider can throw instead of returning an exit status for the copy-back helper, the capture, or the archive download. `remoteWorkspaceFailureCause` then returns `remote_workspace_operation_failed: operation=<op> transport=<label>[ code=<ERRNO>]`. The label is from a closed list (for example `sandbox_lease_inactive`, `permission_denied`, `sandbox_not_running`, `transport_timeout`, `transport_closed`, `daytona_sandbox_not_found`, `provider_error`). Provider text is never copied into the cause. The thrown error does not change, so preparation errors, `classifyNativeWorkspaceFailure`, and other callers keep their contracts. Archive-safety and Git-metadata refusals get no label, because they have their own diagnostics. - `remote-workspaces.ts`: when the last active invocation of a save fails, the save operation becomes "failed" with `exitCode`, `finishedAt`, a redacted `stderrExcerpt`, and `metadata.lastFailure`. A retry of the same boundary sets it to "running" again. A success sets it to "succeeded". After the final save commits, a failed save is retired for export cleanup in the same way as a running save. - `remote-workspace-snapshot-retention.ts`: a failed save that is not retired keeps its snapshot pinned, as it did while it was "running". Its retry can still use an uploaded snapshot. - `native-run-finalizer.ts` and `native-workspace-finalizer.ts`: the redacted cause of each copy-back attempt goes to the coordinator `failureDetail.failureCause` and the run `resultJson.failureCause`. When retries stop, the run error adds `Last cause: <cause>`. The server checks the cause against a fixed grammar again before it writes it. A cause with a path or other text is dropped. - `native-post-result-failure.ts` (new), `native-session-executor.ts`, `runtime-execution.ts`: a failure after an accepted result gets the label `runner_settlement_incomplete` when the runner did not prove a durable suspension. The failed `workspace_finalize` operation records this closed diagnostic as `failureDiagnostic`. The coordinator records it as `native_post_result_failure: category=<label>`. The control code `native_session_lease_lost` and its fencing do not change. ## Verification - Red, before the fix (the new tests run against `master` source): - 4 provider-refusal cases and the archive download failure: no redacted cause was available, so the copy-back recorded only the bare code. - Failed save, with only the transport fix applied: `- "status": "failed"` / `+ "status": "running"`. - Run cause after retries stop: `expected null to be 'remote_workspace_operation_failed: …'`. - Green, after the fix: `cd server && npx vitest run src/services/remote-workspace-transport.test.ts src/services/remote-workspace-save-concurrency.test.ts src/__tests__/native-workspace-finalization-work.test.ts src/services/native-runtime/native-post-result-failure.test.ts src/services/remote-workspaces.integration.test.ts src/services/remote-workspace-snapshot-retention.test.ts` passes. - Safety cases: - The failed save is retried on the same boundary operation, records "succeeded", and keeps the saved file bytes. - `daytona_sandbox_not_found` is still classified as `workspace_sync_out_unrecoverable`. - An archive-safety refusal keeps its exact text. - A forged cause with a path never reaches the run or the coordinator. - A run-level timeout cause (`remote_workspace_operation_timed_out: capture (configured timeout 300000ms)`) reaches the run and the coordinator. - `heartbeat-native-remote-failure-retention.test.ts` passes: a preparation failure keeps its exact run error. - The existing integration tests for retired saves, poisoned reservations, duplicate invocations, and snapshot retention pass. They prove export cleanup and identity protection. - `tsc --noEmit` for `server` reports no errors. - Not done: a live Daytona copy-back. This PR adds diagnostics and corrects the operation state. It does not change when a retry runs. ## Risks - Thrown copy-back errors do not change. Only the recorded cause is new. The `daytona_sandbox_not_found` and archive-safety classifications are tested. - A failed save operation is now "failed" before the final save commits. Retirement, export cleanup, and snapshot pinning now also select these failed rows. They select only rows with `lastFailure` and no retirement receipt. - Privacy: each new cause is a closed label or a server-built code. The server checks it again with an anchored grammar before it writes the run. No sandbox output, path, or message text is stored. - The retry count, the retry delay, and the classification of each failure do not change. No provider turn is added. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Provider: Anthropic. Model: Claude Opus 5.5 (`claude-opus-5-5`), in the Claude Code agent harness. - Capabilities used: tool use, shell and code execution, repository search, read-only browser reads of staging, and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Nicky Leach
|
8c062fa481 | fix(native): deliver plugin VM Runner usage to Core (#19162) | ||
|
DottaandClaude Opus 5.5
|
512fbea0bb |
fix(native): deliver messages saved while a native run finalizes (#19183)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A user can reply on a task while the agent's run is still finishing > - A remote native run first commits its result and its status decision (for example `done`). Then its sandbox environment settles (harness backup, warm retention). > - A message that arrives after the final turn took its input is saved as a deferred receipt behind that run > - The saved-message recovery path holds the receipt while the source run still owns its environment lease or its workspace copy-back owner > - Nothing notified that path when the lease or the owner settled, so the messages waited for an unrelated company event > - This pull request notifies the existing deferred-receipt work when a run's environment lease settles or its workspace finalization owner is released > - The benefit is that messages sent while a run finalizes are delivered once, soon after the run settles, and a task closed in that window is reopened for them ## Linked Issues or Issue Description **What happened?** A user posted three plain comments on a task while the agent's remote native run was finalizing. The final turn had already called its finish tool. The comments were saved in one deferred receipt. The run then committed `done` and its status became `succeeded`. No follow-up run started. The task stayed `done` with three unanswered requests. About 20 minutes later, an unrelated event in the same company woke the deferred-receipt worker, and the messages were delivered. **Expected behavior** A message that arrives after the last turn took its input, but before the run releases, gets one follow-up turn as soon as the run settles. If finalization marked the task `done` in that window, the task is reopened for the message. Messages that the finished turn already consumed are not replayed. **Steps to reproduce** 1. Start a native run with a remote (sandbox) environment. 2. After the agent calls its finish tool, post a user comment on the task. 3. Let finalization commit `done` and project the run as `succeeded`. 4. Observe that the comment's deferred receipt stays `deferred_issue_execution` after the environment lease settles. **Root cause** A remote native run is finished by native finalization, which projects the terminal status. The in-process run completion (`releaseIssueExecutionAndPromote`) does not run for it. Its saved messages depend on the stranded-queue sweep in `resumeDeferredExecutionReceipts`, which calls the release with `resumeStrandedQueue`. That release correctly waits while `hasPendingRunCleanup` is true (an unsettled environment lease or a workspace finalization owner). Finalization notifies the deferred-receipt worker when it projects the run, but the cleanup is still pending at that time. The worker then drops the entry. Lease settlement and owner release did not notify the worker again. Refs #19122 (saves a redirect sent while a stopped run settles; this change reuses the same deferred-receipt delivery path for normally completing runs). ## What Changed - `environmentService.releaseLease` notifies deferred-execution work for the company when a run's lease leaves `active` or `pending_cleanup` (released, retained, or failed). - `environmentService.releaseLeasesForRun` notifies deferred-execution work in the same way. - `withNativeWorkspaceFinalizationOwnership` notifies deferred-execution work when it releases its owner. - Delivery uses the existing path: `resumeDeferredExecutionReceipts` → stranded-queue release → deferred wake promotion (with the existing reopen rule for user messages). No new delivery mechanism. - New regression in `deferred-execution-receipt-work.test.ts` with the real deferred-receipt worker, the real release adapter, and the real lease service. - `doc/execution-semantics.md`: one additive paragraph for messages sent while a run finalizes. ## Verification - Staging evidence (read-only): the task's saved queue held the three comments as one `deferred_issue_execution` receipt after the run succeeded. The run had no `run succeeded` lifecycle event (it was finished by native finalization, not run completion). The receipt was promoted about 20 minutes later with `reopenedFrom: done` by the same recovery path. - Red before the fix: the new test failed with `expected null to be truthy` (the receipt stayed deferred with no run after the lease settled). - Green after the fix: `cd server && npx vitest run src/__tests__/deferred-execution-receipt-work.test.ts -t "finishing native run"`. - Safety cases in the new test: - The receipt stays deferred while the environment lease is active (cleanup holds still apply). - After settlement, exactly one run carries both messages in order, and the done task is reopened once. - A second settlement signal does not deliver the messages again. - Related suites: `deferred-execution-receipt-work`, `environment-service`, `environment-cleanup-work`, `native-environment-settlement-work`, `idle-sleep-safety`. - Not done: a live staging reproduction at the fixed revision. Deployment is out of scope. ## Risks - The deferred-receipt worker reloads the company's deferred receipts after each run lease settlement. Each reload is bounded and runs only when deferred receipts exist. The notification is transactional, so a rolled-back release does not wake the worker. - Low risk: no schema change, no API change, and no change to admission or reopen rules. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file edits, browser for read-only staging diagnostics). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
a0250284e3 |
fix: preserve proposal revision turns and prove chat completion (#19105)
## Thinking Path > - Paperclip manages work done by AI agents. > - A completed delegated task must report its result to the originating chat. > - That report uses a real provider turn and must stay in cost accounting. > - Ordinary first-task proposals must also support the promised chat acceptance path. > - A revision response can reject an older proposal before it creates the replacement. > - The old rejection transition clears that active response's ownership and blocks its next tool. > - This change proves required completion work, guides ordinary proposal policy and preserves the revision turn under the existing locks. ## Linked Issues or Issue Description No separate issue exists. The original failures remain in [the complete Product run](https://github.com/paperclipai/paperclip/actions/runs/41915965118). The independent continuation repair is #19076. Related #12209 concerns prior-run confirmation binding. This change fixes the ownership reset during rejection, rather than changing binding authorization. **What happened?** The complete 20-case run finished with 13 passes and 7 failures. Three delegated-chat cases counted one required `chat_task_completed` provider turn as unexpected work. Two first-task agents chose optional human-only cards, then correctly received 403 when they tried conversational acceptance. A [reviewed two-case candidate run](https://github.com/paperclipai/paperclip/actions/runs/41931445311) then used the changed first-task skill. Both models inherited the ordinary resolver policy. Clarify/propose/accept passed all 12 predicates. Revise/accept failed after the new response rejected its predecessor's proposal. The saved rejection was followed by `paperclip_runner_tool_binding_not_authorized` on its next tools. The final `native_session_lease_lost` label followed those binding failures; this PR does not assume lease expiry caused them. Both cells cleaned up and published their original grades. **Expected behavior** Keep the existing ordinary-turn bounds and account for one independently proved completion notification. Inherit the normal resolver policy for ordinary proposals that accept chat replies. When an active response rejects an older proposal, save the rejection and retain that response's current ownership so it can create the revised proposal. Keep human, current-source and native completion-review rejection behavior and all real approval restrictions. **Steps to reproduce** The original full run used `4740d251`. The two-case candidate used trusted workflow `7d1e9b34` and target `74da56c4`. Run the focused provider-free controls below. The dual-run API regression first fails because rejection changes the issue to `todo` and clears the newer run's execution and checkout fields. ## What Changed - Prove one completion notification only for native Codex/local `multi-repository`, `clarify-reuse` and `plan-handoff`. Keep the ordinary bounds of 2, 3 and 4. Preserve actual totals of 3, 4 and 5, all run IDs and all usage. - Read those declared totals from the current catalog, which now includes the completion reply. Check the total once and prove that exactly one turn is the required notification. Do not add another turn while waiting. The required-server controls use the actual catalog tasks. - Bind the notification to the same company, agent, conversation generation, completed child, single delivery, accounted native run, saved reply and rendered result link. Reject extra work, duplicate deliveries, unknown events and mutations, including native work without a Paperclip tool call. - Collect workspace operations through the real run API for these three proofs. Keep other collector calls unchanged. Keep the proof controls in the existing required server test lane. - Stop idle polling on a permanent completion-proof error. Continue to retry transient observations within the original 240-second window. - Guide ordinary chat-accepting first-task proposals to omit optional `resolverPolicy` and inherit server/company defaults. Preserve explicit and governed human-only restrictions. Do not lower or recreate saved restricted cards. - Under the existing locked issue and interaction rows, preserve the active responding run when it rejects a different source run's proposal. Change only the ordinary requeue predicate. Keep the saved rejection, notifications, activity and existing Telemetry calls. Keep global status, lease and tool-authority guards unchanged. - Extend the existing native API controls with the actual dual-run rejection/revision sequence and policy, provenance, human, current-source and native completion-review countercontrols. No new workflow or test framework is needed. ## Verification - Current combined head is `a8567ded67e5ab811675390e9a5c5ed2194aee2f`, integrated cleanly on `fb96537e28c4c7bfced4939909cb334392320868`. The upstream completion-reply waiter is preserved. [Current required CI](https://github.com/paperclipai/paperclip/actions/runs/41940780852) passes. Actual test checkout `ff93c034d7bb40f3df7bc75cf51828fd07bb0d29` merges this head into `9c6993f0aab7ebb22a26d78f5d227accb35a6ad1`. - Fresh Greptile is 5/5 on exact head `a8567ded`, with no blocking finding. All addressed review threads are resolved. The separate automated reviewer emitted a paused-work response; that response is not a review approval. - Current required server CI actually passed all 11 native confirmation API cases, 46 completion-proof controls and 13 onboarding asset checks. Canonical typecheck, Build, the native package/server suites, all browser and serialized shards, `ci / e2e` and `ci / verify` pass. No current-head CI retry was needed. - Local dual-run RED: the unmodified transition returned HTTP 200 and saved rejection, then changed the issue to `todo` and cleared the newer run's execution/checkout fields. Whole setup and check took 28.95 seconds; the case took 596 ms. - Local treatment: all 11 real native/API cases pass. The same-session response retains execution/checkout ownership, reads task context and persists the recorded-shape revised confirmation. Its stored lease receipt stays intact and one rejection activity is saved. Human-only rejection, current-source rejection, human rejection with/without a newer owner, Plan/Ask, foreign provenance, authentication and formal approval controls pass. - All 59 existing confirmation resolver controls pass in 16.62 seconds. These cover cancellation, lost ownership, company/user scope, policy changes, transactional rollback, concurrent retries and stale revisions. - All 46 completion controls pass in 4.07 seconds. The two added clock controls preserve the original idle deadline and prove immediate permanent-proof failure plus transient observation retry. The existing live collector-to-proof and native mutation negatives remain in this required server file. - The first current-catalog integration check failed 5 controls in 2.70 seconds. The old consumer added the already-declared reply again and could accept an extra ordinary turn. The corrected consumer passes all 46 controls using the actual catalog. The existing full 20-case catalog/reporting compatibility check also passes in 4.93 seconds. Case identities, configured models and reporting match the declared 20. - The API controls used one owned loopback PostgreSQL instance through existing supported test hooks. It was stopped and removed. Earlier missing-library/setup failures and one corrected current-source test expectation remain retained; they are not counted as product failures or passes. - The unchanged first-task skill has SHA-256 `b789090996e7925676590c006418eb8b57f4514e646f6bca18ffd7b2b8e864de`. Both candidate runtime copies matched it. The inherited policy was observed in both models; one complete acceptance journey passed. This does not qualify the new rejection repair. - The three retained real completion streams pass the stronger local proof, including native items, workspace observations and recorded DOM links. Their original failed grades stay unchanged. - Earlier related checks passed 153 hiring/non-execution controls and 13 asset checks. Their source is unchanged. Required discovery includes the completion, API and asset files exactly once. - Prior candidate `74da56c4` passed all required CI, including canonical typecheck and build, in [CI run 41929148958](https://github.com/paperclipai/paperclip/actions/runs/41929148958). Its fresh Greptile was 5/5. This is prior-head evidence. - The intermediate `053150d9` head also received Greptile 5/5. Its [CI run](https://github.com/paperclipai/paperclip/actions/runs/41940014165) and review stay retained as superseded evidence. They do not qualify the final catalog correction. - A clean full local typecheck/build is not claimed because the reused dependency cache has stale adapter-utils/Runner declarations. No source or lock workaround was made. Fresh canonical CI must pass before merge. - The original full-plan result remains 13/20. The separate two-case probe remains 1/2. A fresh complete saved-plan run against delivered source is required for acceptance. No case, assertion, model, budget, timeout or failed grade is changed. Focused commands: ```sh # From server/; database URLs point only to an owned disposable local instance. node ../node_modules/vitest/vitest.mjs run src/__tests__/chat-completion-proof.test.ts PAPERCLIP_TEST_DATABASE_URL="$OWNED_TEST_DATABASE_URL" node ../node_modules/vitest/vitest.mjs run src/services/native-runtime/confirmation-comment-api.integration.test.ts PAPERCLIP_TEST_POSTGRES_ADMIN_URL="$OWNED_TEST_DATABASE_URL" node ../node_modules/vitest/vitest.mjs run src/__tests__/confirmation-comment-resolution.test.ts ``` ## Risks - The ownership exception applies only to an agent acting as the current execution run against a different non-null source run. Human, unowned and current-source requeue behavior must stay intact. Native completion-review handling remains separate. - The completion proof fails closed on new native tools or event shapes. A new shape needs review. - A stored unchanged lease receipt is supporting local evidence. The final real journey must prove full revision, execution and delivery. - The new head still needs normal protected merge approval. The optional queued Paperclip review is not a passing review. No causal repair of unrelated CI intermittency is claimed. ## Model Used OpenAI `gpt-6.1-sol` in Codex, with `max` reasoning effort. The runtime reports a 475,000-token context window. Used reasoning, code editing, shell tools and provider-free controls. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
eab672e10a |
fix(workspaces): name the failing remote copy-back step without private output (#19152)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native run in a Daytona sandbox saves its workspace after the provider turn. The save is called copy-back. > - Copy-back fails on staging after successful provider turns. Issues become blocked with `retry_exhausted`. > - The recorded cause is only `remote_workspace_operation_failed` or `workspace_sync_out_failed`. Operators cannot see which step failed or why. > - The remote command wrapper drops every unrecognized sandbox failure to the bare code. The finalizer retry keeps only the disk-full cause. > - This pull request records a bounded, redacted cause: the operation, its exit code or signal, and the failing capture stage with structured error fields. > - The benefit is that the next copy-back failure shows its first cause, and sandbox output still cannot leak paths or credentials. ## Linked Issues or Issue Description **What happened?** Native runs on a Daytona sandbox finish their provider turn and accept their result. Then workspace copy-back fails. The first `workspace_finalize` operation records only `remote_workspace_operation_failed`. Each retry records only `workspace_sync_out_failed`. The run has no error text. After the retries stop, the issue is blocked. **Steps to reproduce** 1. Run a native task in a Daytona sandbox. 2. Make the sandbox capture step fail with an error that is not disk-full or a known archive validation error. For example, make Git exit with 128. 3. Read the run's workspace operations. **Expected behavior** The failed operation names the step that failed and a redacted cause. **Actual behavior** The failed operation shows only `remote_workspace_operation_failed` and then `workspace_sync_out_failed`. Refs #19050, #19059, #19061, #19063. These are related Daytona and finalizer repairs. This PR does not duplicate them. ## What Changed - `remote-workspace-snapshot-archive.ts`: add `remoteWorkspaceFailureDetail`. It is self-contained and runs in the sandbox. It reports only fixed keys (`code`, `exit`, `signal`, `syscall`, `killed`, `program=git`, `name`, `reason`). Each value is at most 64 `[A-Za-z0-9_]` characters and comes from structured error properties. It never reports message text, command lines, or paths. The archive child now writes `remote_workspace_archive_failed: <detail>` for an unclassified capture error. - `remote-workspace-transport.ts`: the helper-install and capture scripts write `remote_workspace_capture_failed: stage=helper|archive|publish <detail>`. The server accepts this text only when it matches a strict grammar. Unrecognized failures now throw `RemoteWorkspaceOperationError` with the message `remote_workspace_operation_failed: operation=<op> exit=<n>[ signal=<SIG>][; <validated cause>]`. Existing exact diagnostics (Git operation in progress, full disk, archive validation) do not change. - `native-workspace-finalizer.ts`: a retried remote save records the redacted cause in the operation stderr and in `metadata.workspaceSync.cause`. Other error text still stays off the record. - Tests: new red/green cases. The old privacy tests now assert the anchored redacted form, which still excludes any synthetic private text. ## Verification - Red, before the fix (only the error class existed). 8 tests failed on behavior: - The real Git failure produced `Received: "remote_workspace_operation_failed"`. - The signal-terminated command produced the same bare message. - The forged-cause cases produced the same bare message. - The finalizer retry case lost the cause. - Green, after the fix: `npx vitest run src/services/remote-workspace-transport.test.ts src/services/native-runtime/native-workspace-finalizer-failure.test.ts src/services/remote-workspace-snapshot-archive.test.ts` reported 126 passed. - The new capture test runs the real helper scripts with a controlled `git` that exits 128 and prints a synthetic private path. The error is exactly `remote_workspace_operation_failed: operation=capture exit=1; remote_workspace_capture_failed: stage=archive exit=128 program=git helper_exit=1`. The private path is absent. The workspace files are unchanged. The test passes with split output and with one-shot output. - Forged causes are dropped. These cases are tested: a path in a value, an extra line, an unknown key, and a prefixed text. - Wider sweep: I ran all 63 server test files that touch remote workspaces, copy-back, or finalization. 6381 tests passed. 2 tests failed in `chat-channels.integration.test.ts` and `git-workspace-expiration.test.ts`. These 2 tests also fail with this PR's three source files reverted to `master`. This PR does not cause them. - Typecheck: `tsc` reports no errors in the changed files. Other errors are local, from unbuilt workspace packages. - Not done: I did not run a live Daytona copy-back. This change adds diagnostics. It does not repair the underlying failure. The staging failures can be diagnosed after this PR is deployed. ## Risks - The error message for an unrecognized remote failure changes from the exact bare code to the same code plus validated fields. No production code compares the exact text. Only tests did, and I updated them. - Privacy: the sandbox detail contains only structured error properties. The server re-validates them with an anchored grammar with a length limit and a field limit. A path, a message text, or an unknown key fails the grammar, and the server drops the whole cause. - Retry and classification policy do not change. `classifyNativeWorkspaceFailure` does not read the new text. - This PR does not address two items: the final snapshot operation stays `running` after retries are exhausted, and no disk-retention cause was proven for the ACPX disk-full report. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, browser read-only GETs on staging, code execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
5b99bdae47 |
fix(native): verify requested publication at finalization (#19130)
## Thinking Path > - Paperclip manages work done by AI agents. > - Native execution reports work completion through a saved result. > - Some native providers can build that result from their final message. > - This path can bypass the finish tool's publication check. > - A task can then be marked done without the document the user requested. > - This change checks publication again at finalization and status commit. > - Missing output uses the existing bounded repair path with specific feedback. ## Linked Issues or Issue Description **What happened?** A native provider returned a final message about a local Markdown briefing. It had not published a task document or a verified attachment. Its synthetic result claimed success. The server accepted the claim and marked the task done. A real PostgreSQL regression reproduced this on public master. **Expected behavior** A requested task document must have valid saved publication evidence before the task becomes done. The check must apply even when the provider does not call the finish tool. A plain answer still needs no document. An explicit document-free fallback requires the actual human rejection for the current unchanged request. **Steps to reproduce** 1. Request a Markdown briefing document on a task. 2. Complete a native turn with a synthetic `provider:opencode:agent-message` result and no published output. 3. Admit the result through the normal control-plane port and finalize it. 4. The old code marks the task done. The repaired code keeps it open and queues one repair with publication instructions. **Paperclip version or commit** Reproduced from public master `7d1e9b3477f12780e987c32fb8191d476c210fda`. Related: #18661 added the finish-tool publication guard. This change applies that guard to finalization. #18180 covers a different fallback problem: no final text. This change preserves native provider fallback generation and its saved claims. ## What Changed - Run the existing deliverable guard before accepting a native done decision. Keep the original provider result unchanged. - Route missing publication through the existing single incomplete-result continuation. Include the publication feedback in the wake. A repeated incomplete result uses the existing named recovery action. - Inspect publication through a typed safe projection of the accepted ledger. Keep whole-protocol validation at result admission. An authoritative done decision must pass publication checks even when its accepted ledger is partial. - Recheck publication under the status-commit transaction. Hold only matching proof rows with non-waiting share locks. Changed or locked proof uses the existing bounded status replanning path. - Permit an explicit document-free decline fallback only for the saved human rejection selected by the current continuation. Require the same original request, current saved description, source revision and source contract. Model prose or an unrelated old decision is insufficient. - Add real database controls for publication, plain answers, authenticated decline, stale or wrong decisions, bounded recovery, and deletion between assessment and commit. ## Verification - The first regression failed before the repair: the synthetic result marked an unpublished task done. The retained real PostgreSQL run took 21.846 seconds, including setup and cleanup. - Two description-edit regressions also failed before their repair: a description change after admission or at the commit boundary still let the old decline waive the newly required document. Both now keep the task open. The red run took 25.738 seconds. - The commit-race regression also failed before its repair: removing publication after assessment still allowed done. That run took 17.448 seconds. - All thirteen focused real database controls pass after the repair. They exercise the normal control-plane port, finalizer, semantic publication tools, saved human rejection and status transaction. Owned database processes, data and output storage are removed after each run. - All 469 tests across the seven relevant and affected suites pass on head `f8870469f9879a53551f0faaff0fe5b1234ea303`, based on master `c7390bef20`. The six-suite run passed 426 tests in 97.319 seconds including owned setup and cleanup. The unchanged reassessment-worker suite passed 43 tests in 48.901 seconds. All required current-head checks pass in [CI run 41937653209](https://github.com/paperclipai/paperclip/actions/runs/41937653209), including aggregate verify, typechecking, build, general and serialized tests, native Runner tests and all eight browser shards. Fresh Greptile review is 5/5 on this exact head, with no open findings. - Local scoped server typechecking reported no errors in the four changed files and 11 errors in reused dependency declarations and exports. That setup failure remains retained. Fresh current-head CI passes the full Typecheck + Release Registry gate and Build. Local typecheck is not claimed as passing. - CI on the previous head found one shared status-projection regression across Studio, routine, status-authority and worker controls. Those phases use opaque accepted ledgers. The correction inspects only publication references, preserves the accepted marker and bindings, and forces the publication check for an authoritative done decision. A new real commit control verifies that an empty accepted ledger cannot close a requested-document task. All original affected assertions remain unchanged. The retained failed run is [41934314437](https://github.com/paperclipai/paperclip/actions/runs/41934314437). - No provider call, paid campaign, staging deployment or eval regrade is part of this PR. No model, timeout, assertion, workflow, dependency or lockfile change. Focused suites: ```sh ./node_modules/.bin/vitest run --config "$PWD/server/vitest.config.ts" --root server \ src/services/native-runtime/paperclip-control-plane-port.test.ts \ src/services/native-runtime/native-completion-feedback.test.ts \ src/services/native-runtime/native-deliverable-feedback.test.ts \ src/services/native-runtime/native-skill-test-status.test.ts \ src/services/native-runtime/native-routine-status.test.ts \ src/__tests__/native-status-arbiter-corpus.test.ts \ src/__tests__/native-status-reassessment-work.test.ts ``` ## Risks A native result that previously closed a task without its requested output now leaves it open. The existing repair path can run once and then records a named recovery action. Database contention does not grant completion. The existing first-task decision gate still runs first. The conditional decline check is deliberately conservative; unmatched wording or missing authoritative source data does not waive publication. This change closes a false completion path. It does not establish why a provider omitted publication, prove a live provider repair, or qualify a full evaluation plan. ## Model Used OpenAI Codex, based on GPT-6, assisted with diagnosis, code and controls. The exact serving model identifier and context-window size are not exposed in this session. Capabilities used: repository inspection, code editing, local code execution and GitHub tools. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal issue identity - [x] I have run relevant tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have documented the behavior and limits in this PR; no operator command or public API changed - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open findings - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandClaude Opus 5.5
|
b9fbbf8d30 |
fix(native): continue runs woken by withdrawn or cancelled interactions (#19163)
Withdrawing a card or cancelling a question wakes the assignee with that interaction id. The native interaction bridge required every wake interaction to be accepted, rejected or answered, so preparation failed with native_interaction_unresolved before the provider started. Project a cancelled interaction as a closed response with its stored result. Consumers grant authority only for accepted, answered or pending items, so the closed response carries context and no approval. Pending requests and governed tool reviews keep their existing checks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Devin FoleyandClaude Opus 5.5
|
7f9ef38273 |
fix(runner): report the upstream cause of OpenCode provider failures (#19166)
OpenCode reports some OpenRouter failures as an UnknownError whose message is the provider's JSON error body. The provider failure now reads its HTTP status, error type and message from that body, so the run shows the upstream message instead of raw JSON. The OpenCode nativeTurnFailure diagnostic now carries the provider failure's category, retryability and HTTP status (closed values only; no provider text or code), and recognizes the driver's opencode_empty_response code. Run status, error codes and retries do not change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Nicky Leach
|
737db73275 | fix(native): measure resumed run usage from the prior run's total (#19161) | ||
|
Devin FoleyandClaude Opus 5.5
|
7edbc1fc07 |
fix(server): keep a successor's issue lock when a finalizing native run releases its checkout (#19156)
`release_checkout` now clears `checkoutRunId` and the execution lock only when they belong to the committing run, so a successor that claimed the issue while its predecessor was finalizing keeps its claim and passes reviewed-chat attestation. `attestReviewedExternalChatRun` reports the first failed check as a closed label. Setup failures store it in `resultJson.reviewedChatBinding.denialReason`, and Sentry run-failure diagnostics export it as `execution.reviewedChatBindingDenialReason` (no IDs or text). The error message is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
482ec0aea9 |
fix(native): run saved copy-back retries on schedule behind a slow export (#19145)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs copy their sandbox workspace back after the provider finishes. A failed copy-back saves a retry 30 seconds later, with a bound of 3 attempts. > - One shared delivery queue runs these retries. A pass of that queue ends only when its slowest check ends. > - A legitimate sandbox export can take 10+ minutes. While it runs, all other due retries wait. > - On staging, a run active across a deploy showed "Reconnecting" with an overdue retry for about 24 minutes. Its retries ran at +24 min and +3.5 min instead of +30 s. It settled only after about 32 minutes. > - This pull request lets idle check slots admit entries that come due during a pass. It does not overlap a running check's resource and does not start work after abort or drain. > - The benefit is that the existing 3-attempt bound settles a failing copy-back in about 1 minute, not after the slowest unrelated export. ## Linked Issues or Issue Description **What happened?** A native run whose result is durable but whose workspace copy-back failed stays in `running` with the label "Reconnecting" for a long time. Its saved retry time is in the past. The retry does not run until an unrelated slow copy-back in the same queue pass finishes. **Steps to reproduce** 1. Start two native runs on different tasks with accepted results. 2. Let the copy-back of run A take many minutes. 3. Let the copy-back of run B fail. It saves a retry 30 seconds later. 4. Watch run B. Its retry does not run until run A's copy-back ends. **Expected behavior** Run B retries on its saved schedule. After 3 failed attempts it settles with `native_workspace_sync_out_retry_exhausted` and a recovery action, while run A continues. **Paperclip version or commit** master at |
||
|
|
c2cc69d1e4 |
fix(recovery): resume and settle preserved native results (#19034)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native execution can save an accepted result before workspace export fails. > - Recover must preserve the files and settle that same result without another agent turn. > - The export failure can own the task's Blocked transition and its recovery notice. > - Recovery must prove preservation, stop, source identity and current authority before settlement. > - Recover can also transfer repair ownership to the system, which must keep explicit board export repair available. > - This change restores those paths while preserving later task state, live owners and unknown action outcomes. ## Linked Issues or Issue Description Related #17916 protects unfinished exports. Related #19004 preserves deliberate Retry authority. This change repairs accepted-result finalization and its recovery notice. **What happened?** A native run accepted its result, then exhausted workspace export. Normal Recover could save the files but leave the result and old recovery notice unsettled. A saved Retry could also use a different generation from preservation. Separately, Recover changed the action owner from board to system. Explicit export Retry then rejected that same source as stale, even when its accepted result and stopped allocation still matched. **Expected behavior** Preserve files, finish the original accepted result and close only its obsolete notice after exact proof. Keep explicit board export repair available after system adoption. Preserve current authority, Stop, later task changes, newer execution and every physical ownership fence. **Steps to reproduce** 1. Accept a native result and exhaust its workspace export. 2. Request ordinary Recover, optionally with an existing saved Retry. 3. Let file preservation complete. Inspect the task and source recovery notice. 4. To reproduce the explicit-repair defect, call export Retry after Recover adopts the action. Before this repair it rejects the valid system-owned source before any provider call. ## What Changed - Record the exact accepted result, contract, task status version and prior decision when export exhaustion creates Blocked. - Bind passive finalization to complete, source-bound preservation and the current chat/review authority. Recheck the proof under status commit locks. - Keep the saved Retry identity separate from its preservation generation. Retain coalescing and reject malformed or mismatched history. - Preserve an existing published GitHub assessment for the no-publication status-unproven wait. Keep all semantic, result, authority and publication checks. - Close an old export notice only after its own accepted result, export or preservation receipts, effects, projection and old Runner ownership settle. Keep newer work and files unchanged. - Permit explicit board export Retry after Recover transfers the exact source to the system. Require valid system and workspace recovery records for that same run, plus every existing result, stop, allocation, task and shared physical-lock check. Reject missing proof, another source and agent ownership before provider access. - Merge current master while retaining the stronger Stop, source and verified preservation checks. - Update the execution contract. No schema, endpoint, Runner protocol, model, trust or provider artifact change. ## Verification - Current head: `9be5a73489801095c9f39c9bfcb285a22b055912`. - The explicit-repair regression fails before the new guard change: ordinary Recover adopts the real database action, then export Retry rejects it as stale. It passes after the repair. - On the merged source, three owning PostgreSQL/file suites pass **220 tests**: `preserved-native-response-wait.integration.test.ts`, `native-workspace-export-retry.test.ts` and `idle-completed-history.test.ts`. The export suite includes four unrelated-owner/source rejection controls and both board-owned and system-adopted active-exporter controls. Results and run counts prove that repair creates no provider turn. - Server layered typecheck, module boundaries, `git diff --check`, and the added-line secret/private-incident scan pass. - Full workspace typecheck (`pnpm -r typecheck`) and build (`pnpm build`) pass. The complete local `pnpm test:run` is still running; its completed serialized chat partitions pass. Current-head CI is green (53 successful checks, 3 expected skips). Greptile reviewed `9be5a73489801095c9f39c9bfcb285a22b055912` at 2026-11-05 00:58 UTC with confidence **5/5**; all review threads are resolved. - Original qualification remains supporting evidence: the accepted-result fixture exercises HTTP Recover, real files, the finalizer, saved Retry coalescing, eight commit races, current principal/resource/head changes, Stop, later task changes, notice closure and repeated delivery. No accepted response, provider turn or publication is replayed. - Earlier failures remain recorded. Original red controls exposed wrong-action preservation, unrelated finalization barriers, generation mismatch and a lost GitHub assessment; the original repair added controls for them. The prior CI run [41908572248](https://github.com/paperclipai/paperclip/actions/runs/41908572248) failed an unchanged completed-history runtime fixture because its loopback service refused connections through the original ten-second readiness window. That owning suite passes unchanged in the 220-case run above. The earlier readiness failure's cause remains unproven; no wait or assertion was weakened. - Local and CI tests do not establish hosted task recovery. The serving revision and original-task outcomes require separate verification after rollout. ## Risks - Historical failures without the exact export-owned Blocked receipt keep their current task status. A saved result does not grant execution or publication authority. - Explicit export repair still resumes only the same retained allocation under the shared physical lock. It cannot take over an active exporter, create another agent turn, choose a replacement sandbox or discard saved files. - Retiring a source notice does not release physical resources or modify a newer task owner, later Done state or saved snapshot history. - The proof adds bounded reads only for concrete recovery obligations. It adds no recurring worker or polling timer. ## Model Used OpenAI GPT-6 through Codex for the original repair. OpenAI GPT-6.1 Sol (`gpt-6.1-sol`), xhigh reasoning, for the merge, integration verification and system-adopted export repair. Both used repository tools, code execution and PostgreSQL tests. The runtime does not expose a context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used - [x] I have checked ROADMAP.md and searched for duplicate or related work - [x] I have described the issue and linked related public PRs - [x] I have included no internal task identifiers or private incident links - [x] I have added behavioral and safety regression tests - [x] I have documented verification limits and risks - [x] I have updated the execution contract - [x] All Paperclip CI gates are green on the current head - [x] Greptile is 5/5 on the current head with no unresolved comments - [x] I will address review comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Devin Foley <devin@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
11f2a5293d |
fix(native): name the proof that holds a recovered per-turn owner (#19061)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native Runner per-turn sessions in a remote sandbox keep one conversation across task turns. > - A follow-up must prove that the previous Runner owner is finished before it starts a new Runner. #18846 added a retirement path for an owner that finished without a saved Runner checkpoint. > - On a test deployment, follow-ups still fail with `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate`. The retirement path has about twenty separate fences, and each refusal gives the same error. > - The read-only run data cannot tell which fence held these runs. A code fix without that evidence would be a guess. > - This pull request names the first failed retirement proof in the error. Admission behavior does not change. > - The benefit is that the next failure on a deployment identifies the exact fence, so the follow-up fix can target it. ## Linked Issues or Issue Description Refs #18846 (retirement of recovered per-turn owners). Refs #18992 (settlement from live state). **What happened?** On a test deployment, 16 runs in one day failed before provider work with `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate`. All were per-turn `acpx_runtime` runs in a Daytona sandbox, and each got a new sandbox. In the runs that I inspected, the previous run committed its result. In one of them, `harness_state.backup.persist` and `session.checkpoint.persist` ran for about 346 seconds and failed, so the Runner left no suspension receipt or failover backup. That is the case that #18846 addresses, but the follow-ups still failed. The run log does not show the retirement message, and the error is the same for each fence. Thus the cause cannot be found from the run data. **Expected behavior** When admission holds a follow-up, the error names the proof that failed. **Steps to reproduce** 1. Seed a per-turn sandbox owner that committed its result without a Runner suspension receipt. 2. Make one retirement proof fail. For example, leave the owner phase `observed`, or leave a `runner.suspend` command pending. 3. Admit a follow-up in a new sandbox. 4. Before this change, every case fails with the same error text. **Paperclip version or commit** Observed on a deployment before `e107a75755`. The same error text is on `master`. ## What Changed - `retireFinishedPerTurnRunnerdOwner` returns the refusal reason instead of `false`. `readSettledRunnerdProjection` returns the journal refusal reason instead of `null`. - When admission tries the retirement and it fails, the error is `prior_owner_terminal_state_indeterminate; retirement_refused=<reason>`. A warm session gets no suffix, because retirement does not apply to it. - Reasons use only fixed lists of known names: owner phase, recovery state, controller fence reason, unsettled command type and status, sandbox outcome, and quarantine copy result. Any other stored value appears as `other`. Reasons do not include payloads, file names, or paths. - The same proofs apply in the same order. Admission accepts and refuses the same states as before. - `doc/execution-semantics.md` describes the suffix. ## Verification - **Failure before the change:** I ran the updated `native-recovered-owner-followup.test.ts` against the unchanged executor. 18 tests failed. Example: expected `...indeterminate; retirement_refused=owner_observed`, received `...indeterminate`. - **After the change:** `cd server && pnpm exec vitest run src/services/native-runtime/native-recovered-owner-followup.test.ts`: 22 passed. Each held case now asserts its reason: `owner_retryable_failure`, `controller_remote_controller_lease_live`, `owner_recovery_resuming_session`, `journal_commands_unsettled(runner.suspend.pending)`, `journal_tool_input_unanswered`, `quarantine_journal_identity_mismatch`, `runner_lifecycle_not_suspended`, and `quarantine_runner_lifecycle_not_suspended`. - The success cases still retire the owner and admit the follow-up. - `pnpm exec vitest run src/services/native-runtime/native-session-executor.test.ts src/services/native-runtime/native-runner-restart-recovery.integration.test.ts src/services/native-runtime/remote-warm-session-recovery.test.ts`: 1227 passed, 20 skipped. - `pnpm exec vitest run src/services/native-runtime/native-controller-recovery.integration.test.ts src/__tests__/task-workspace-binding.test.ts`: passed. - `tsc --noEmit` reports no errors in the changed lines. This worktree did not build every workspace package, so the full-package typecheck shows unrelated missing-module errors. CI runs the full typecheck. - **Not verified live:** no deployment has run this change. ## Risks - Low risk. Admission logic is unchanged. Only the error text gets a suffix. - Consumers that match the error by prefix still match. `failed-run-retry.ts` uses `startsWith("runner_state_identity_mismatch: ")`. No code compares the full string outside tests. - This change does not fix the held runs. It finds the fence. A second change is needed after a deployment reports the reason. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use, shell execution, and read-only browser access to a test deployment. The harness does not expose the context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
3a72efebe5 |
fix(native): confirm no runner process when remote staging fails before launch (#19059)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native sandbox runs supervise a remote `paperclip-runnerd` process. When that process exits during a turn, the transport restarts it inside a reconnect grace window. > - The server launcher stages the runtime assets first. Then it sends one launch RPC that starts runnerd. > - When staging failed, the launcher reported the exit as unconfirmed. This happened although no launch RPC was sent and no process could exist. > - Reconnect recovery treats an unconfirmed exit as a possible second owner. So it stopped at once with `runner_process_termination_unconfirmed` and did not retry. > - This pull request confirms the exit for failures before the launch RPC. Failures after dispatch stay unconfirmed. > - The benefit is that a transient staging failure, such as a full sandbox disk, uses the normal bounded retry. A real failure ends with `transport_reconnect_grace_exceeded` and keeps the staging error as the detail. ## Linked Issues or Issue Description **What happened?** On a hosted staging deployment, 7 native ACPX runs failed in 24 hours with `Provider notification transport failed closed.` (`native_provider_terminal_failed`). In every run, the runner process exited during the turn. The replacement launch then failed while it uploaded the runtime context into the sandbox: `runner_process_termination_unconfirmed: replacement runner health could not be verified: ... .paperclip-upload-<uuid>.tar.gz: create: ... no space left on device`. The sandbox disk was full. No replacement process was started. But the launcher said that it could not confirm the exit. So recovery did not retry inside its grace window. **Expected behavior** A staging failure before the launch RPC proves that no replacement process exists. Recovery must retry within the configured reconnect grace. If the failure continues, the run must end with the reconnect-grace failure and the staging error. **Steps to reproduce** 1. Create a remote runner launcher whose `ensureArtifact` step throws, for example with an ENOSPC upload error. 2. Wait for the handle completion to reject. 3. Before this change, `isExitConfirmed()` returns `false`, and no launch RPC was sent. The transport then fails closed with `runner_process_termination_unconfirmed`. **Paperclip version or commit** Observed on staging revision `89703c3`. Reproduced on `master` at `be20d4ad97`. Related: #19035 and #19014 change other runner paths (replayed terminal turns and redacted external tool results). They do not change remote launcher exit evidence. #18999 adds a free-space reserve for host-volume seed copies. It does not change sandbox disk use or this classification. ## What Changed - `createRemoteRunnerProcessLauncher` records when it dispatches the launch RPC. If the completion fails before that point, the handle reports `isExitConfirmed() === true`. The original error is kept unchanged. - A failed or timed-out launch RPC still reports an unconfirmed exit, because the RPC may have started runnerd. - Added a launcher regression for a staging failure before dispatch. - Added safety cases for a failed and a timed-out launch RPC. - Added a transport regression with the real `paperclip-runnerd`. A replacement whose staging fails is retried, and the next launch restores the durable session without a notification failure. ## Verification - Red before the fix: `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts -t "remote runner process supervision"` failed `confirms no replacement process exists when staging fails before the launch RPC` with `expected false to be true`. The two launch-RPC safety cases passed. - Green after the fix: `npx vitest run src/services/native-runtime/native-session-executor.test.ts` passes 1131 of 1131 tests. - `npx vitest run src/services/native-runtime/setup-readiness.test.ts` passes 98 of 98 tests. - `cd packages/paperclip-runner && npx vitest run src/live/runnerd-codex-transport.test.ts` passes 246 of 246 tests with the real debug `paperclip-runnerd`. This includes the existing case where a replacement with a dispatched but unverifiable process still fails closed. - `npx tsc --noEmit -p server` and `pnpm --filter @paperclipai/paperclip-runner typecheck` pass. - Not run: a live hosted sandbox with a full disk. The staging data showed the exact error, but this change was not deployed there. ## Risks - Low risk. The change affects only failures that occur before the launch RPC. In that window, the launcher has not run any sandbox command that can start runnerd. - Retries now run for the reconnect grace window, which is bounded and uses the existing backoff. Each failed staging attempt cleans its own upload scratch. - This change does not free sandbox disk space. If the disk stays full, the run still fails, now with `transport_reconnect_grace_exceeded` and the ENOSPC detail. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use, shell execution, and code editing. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
5723fd82ca |
fix(native): release a fresh claim when task preparation fails (#19065)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native run claims its finalization coordinator before task preparation. Then it starts the provider turn. > - Task preparation can fail after the claim and before any provider turn. > - In that case the executor kept its claim. Failure settlement read the claim as a live owner and stopped. Nothing failed the run. > - The 60-second execution-control watchdog then failed the run as `execution_finalization_deadline_exceeded`. The original error was lost. > - This pull request releases only that exact fresh, unstarted claim after renewal stops. > - The benefit is that ordinary settlement fails the run at once with its real error and keeps the workspace files. The watchdog stays a backstop. ## Linked Issues or Issue Description Refs #19028. That PR keeps the earlier failure code when the watchdog fails a run. In this case no earlier code is saved, so it cannot help. This PR removes the wait for the watchdog. I searched open PRs for duplicates. #18820, #19034 and #19038 touch near code, but none release a preparation claim. **What happened?** A native task with a prepared sandbox workspace failed during preparation, after the controller claim committed. The coordinator stayed `observed` and kept this controller's lease owner. The heartbeat failure path calls workspace preservation. Preservation refuses any lease owner without a matching restart claim, so it raised `NativeWorkspaceFinalizationBusyError`. The heartbeat then treated ownership as held and returned without a terminal write. About 60 seconds later the watchdog failed the run with a generic "verify the provider stopped" action. The real error (for example `native_harness_storage_owner_missing`) stayed only in the workspace operation record. **Expected behavior** When no provider turn started under a fresh claim, settle the failure like a failure before the claim: retain the source, fail the run with its own error, and do not wait for the watchdog. **Steps to reproduce** 1. Start a native run with a prepared remote workspace. 2. Make task preparation fail after `native.coordinator.claim` commits (for example, a preparation callback throws). 3. See that the run stays `running` until the control deadline fails it as `execution_finalization_deadline_exceeded`. ## What Changed - `native-execution-lease.ts`: add `releaseUnstartedNativeExecutionClaim`. It clears the lease owner and expiry only for the exact run, company, issue, owner and attempt, while the phase is `observed` and no result exists. - `native-session-executor.ts`: when task preparation fails after a fresh claim, mark that claim. The startup owner releases it after renewal stops. A restart-recovery claim, a startup-cleanup context, Stop, restart detach, or a different owner keeps the old behavior. - Tests: an executor regression for the fresh and restart cases, and a real-database test for release and workspace retention against the exact owner, a foreign owner, a different attempt and an accepted result. - `doc/execution-semantics.md`: state that the deadline is a backstop and describe the release. ## Verification - Red before the fix: `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts -t "releases only a fresh claim"` fails with "releaseUnstartedNativeExecutionClaim ... Number of calls: 0". - Green after the fix, with the same command. - `npx vitest run src/services/native-runtime/native-session-executor.test.ts src/services/native-runtime/native-failed-workspace-recovery.test.ts src/services/native-runtime/native-execution-lease.test.ts src/services/execution-control-reconciliation.test.ts` passes (embedded Postgres for the retention test). - `pnpm --filter @paperclipai/server typecheck` passes. - Not done: a live run against a real sandbox at this revision. The staging evidence is read-only. ## Risks - The release is fenced by the exact owner, attempt, `observed` phase and no result. Renewal stops before the release, so a renewal cannot see a missing owner and detach or cancel the run. - The fix covers failures inside task preparation only. Failures later in startup, or after a restart-recovery claim expires, still reach the watchdog. Those keep fail-closed behavior and need separate evidence. - If the release write fails, the run keeps the old watchdog path. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
cda9460ae9 |
fix(native): restart remote bootstrap after a shutdown during preparation (#19056)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native runs in a remote sandbox go through workspace preparation
before the Runner process starts.
> - When the server restarts, restart recovery decides if a remote run
can safely restart its bootstrap.
> - A remote run that never started a Runner process, never had a lease
owner, and has no provider evidence can restart its bootstrap. Recovery
calls this an "unclaimed bootstrap".
> - A graceful shutdown (SIGTERM) suspends every running native
coordinator. It sets `leaseExpiresAt = now`, also when the coordinator
has no lease owner.
> - The unclaimed-bootstrap check requires `leaseExpiresAt` to be null.
So a deploy during workspace preparation turns a safe bootstrap restart
into `native_restart_recovery_blocked` /
`ambiguous_remote_recovery_evidence`. The task is blocked and a board
recovery action is created.
> - This pull request lets the check accept an elapsed expiry when the
coordinator has no lease owner.
> - The benefit is that a deploy during remote workspace preparation no
longer blocks the task. Every other contradictory-evidence guard stays
in force.
## Linked Issues or Issue Description
**What happened?**
On a staging instance, three remote-sandbox runs failed with
`native_restart_recovery_blocked` and
`ambiguous_remote_recovery_evidence` after one deploy. Each run was in
native workspace preparation (`snapshot.git`, `pack`, `seed.verify`,
`stage.workspace`). No run had a process start, a process identity, a
provider event, or a checkpoint. The graceful shutdown event was the
only coordinator change. The next server then blocked each run in
restart recovery.
**Expected behavior**
A remote run that was suspended during preparation, before any Runner
launch or lease claim, restarts its bootstrap on the same run. Recovery
does not block the task.
**Steps to reproduce**
1. Prepare a remote native run (prepared native workspace sync, fenced
preparation, no process, no checkpoint).
2. Call `drainRunningRunsForShutdown("SIGTERM")` for the run.
3. Call `claimNativeRestartRecoveries` as a successor controller with
`restartKind: "hard"`.
4. Observe `{ kind: "blocked", reason:
"ambiguous_remote_recovery_evidence" }`. Without step 2, the same run
gets `bootstrap_incomplete`.
**Paperclip version or commit**
Reproduced on `master` at `be20d4ad97`.
**Deployment mode**
Remote sandbox environments with PostgreSQL.
Related: #18877 (remote checkpoint retry reclaim) and #18854
(pre-provider startup retry). Neither changes this predicate.
## What Changed
- `claimNativeRestartRecoveries`: the unclaimed remote bootstrap check
accepts `leaseExpiresAt` when the coordinator has no lease owner and the
expiry has elapsed. A null owner with an elapsed expiry grants no
authority.
- New regression: a remote run suspended by the real graceful-shutdown
drain during preparation is claimed as `bootstrap_incomplete`, with no
recovery action.
- New safety case: an ownerless coordinator with an unexpired lease
expiry still blocks with `ambiguous_remote_recovery_evidence`.
## Verification
- Red before the fix: `npx vitest run
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts
-t "graceful shutdown suspended during preparation"` returned `blocked`
/ `ambiguous_remote_recovery_evidence`.
- Green after the fix: the same test returns `bootstrap_incomplete` with
`providerAttempt: 0` and `controllerGeneration: 1`.
- The existing contradictory-evidence cases (`claimed`, `generation`,
`process`, `launch`, `provider-event`, `checkpoint`,
`invalid-workspace`, `unfenced`) and the new `unexpired-lease` case
still block.
- Full `native-runner-restart-recovery.integration.test.ts` suite and
server `tsc --noEmit` pass locally.
- Not run: a live deploy on a remote sandbox. The fix is not deployed.
The original blocked tasks are not recovered by this change.
## Risks
- Low risk. The change relaxes one field of one predicate. All other
unclaimed-bootstrap conditions stay: prepared workspace, fenced
preparation, attempt 0, generation 0, no controller identity, no lease
owner, no process identity, no session, no checkpoint, no provider
evidence, and no launch or tool event.
- The executor still requires the durable PRP root to prove an
incomplete bootstrap before it starts a fresh bootstrap.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with
tool use (shell, file edits, browser reads of staging). Standard context
window.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|