mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-13 15:55:46 +01:00
3914d0c94a8b7ade00a1a93f275d80f706287be0
7501
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 DottaandClaude
|
3914d0c94a |
test(chaos): add DF1, a sandbox disk-full fault on the final workspace save (#20810)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart chaos harness (`tests/restart-chaos/`) runs real agent
tasks and breaks the control plane on purpose, so failures show up
before staging users see them
> - A census of 48 hours of staging failures found runs whose result
save-back failed because the sandbox disk was full (class F). The
harness never filled a sandbox disk, so it missed that class
> - #20758 made a full-disk save end recovery at once with cause
`sandbox_disk_full` and a Retry. Nothing in chaos exercised that path
end to end
> - A real full disk cannot be made on demand. The owner chose a
test-only hook that makes the save report a full disk, not a small disk
> - This pull request adds the hook at the save's sandbox boundary and a
DF1 fault that arms it, clears it, and grades the result
> - The benefit is that every wave can now check the disk-full rest, the
Retry offer, and that Retry works after space is freed
## Linked Issues or Issue Description
Refs #20758 (product behaviour under test), #20678 (finalization hands a
full-disk copy-back to recovery)
**What is the problem?**
The chaos harness had no fault for a full sandbox disk during result
save-back. Staging saw this failure (census class F in
`doc/plans/2026-11-11-staging-failures-vs-chaos-coverage.md`), and chaos
could not reproduce it.
**What is the expected behaviour?**
A wave can schedule DF1. While DF1 is armed, the task's final workspace
saves fail as a full disk. When DF1 is cleared, saves succeed again. The
task must rest with `sandbox_disk_full` and a Retry, and that Retry must
finish the task.
**Why does it matter?**
Without a fault, a regression in the disk-full path (no Retry, a wrong
cause, or a Retry that does not work) reaches staging users before chaos
sees it.
## What Changed
- `server/src/services/sandbox-disk-full-fault.ts` (new, test-only):
`sandboxDiskFullFaultTarget` wraps the target of a final save only when
the harness armed that save. The capture commands then exit 1 with the
exact message the in-sandbox script writes on ENOSPC, and the provider
download fails with `No space left on device`. The gate follows
`@paperclipai/shared/crash-points`: `PAPERCLIP_DISK_FULL_FAULT_CONTROL`
is read once at import and ignored under `NODE_ENV=production`.
Disarmed, the hook is one comparison and returns the save's own target.
- `server/src/services/remote-workspaces.ts`: the capture call in
`saveRemoteWorkspace` passes its target through the hook. No other
product change. `workspace-recovery.ts` is not touched.
- `tests/restart-chaos/disk-full-faults.ts` (new): the DF1 kind, the
control-file seam (atomic writes), the rest evidence
(`activeRecoveryAction.evidence.systemRecovery.failure.cause.label`),
the `disk_full_fault` T check, and the DF1 R3 rule (one expected Retry
that must finish the task).
- Harness wiring: `--faults DF1` (refused without `--env daytona`, with
a storm, or with PV faults), no scheduled restart, arm at launch, clear
at the first rest just before the Retry click, receipts in
`<wave>/disk-full-faults.jsonl`, and ledger fields `diskFull: {
faultedSaves, restedDiskFull, retryOffered, retryWorked }`.
- Docs: a "Sandbox disk full (DF1)" section in
`tests/restart-chaos/README.md`, and the census plan marks DF1 as built.
## Verification
- `cd server && pnpm exec vitest run
src/services/sandbox-disk-full-fault.test.ts`: the gate is off by
default and under `NODE_ENV=production`, the save's own target comes
back when disarmed, only a listed final save is faulted, the product
capture turns it into the named disk-full failure, and clearing the file
disarms the next save.
- `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts
tests/restart-chaos/disk-full-faults.test.ts
tests/restart-chaos/grader.test.ts tests/restart-chaos/scheduler.test.ts
tests/restart-chaos/provider-faults.test.ts`: planning, the control
file, and grading.
- `node scripts/typecheck-test-projects.mjs tests/restart-chaos` and the
server `tsc --noEmit`: clean.
- Live Daytona waves (Codex, `--storage s3`, image content `96c4339f…`,
run through the two-slot queue):
- `w-20261111T131314-s9411` (B1): the hook was armed in the server but
no save reached it. B1 uses a local worktree synced into the sandbox,
not a remote workspace. Graded `not_applied` as designed. The harness
now requires `--storage s3` and the docs point at R1.
- `w-20261111T131941-s9412` (R1): 3 faulted saves. t01 rested `blocked`
with `sandbox_disk_full`, `user_action_required`, board owner, API
`canRetry: true`. t02 was cleared too early, while recovery still
checked its source; that recovery then saved and the task finished.
Fixed in
|
||
|
DottaandClaude Opus 5.5
|
0b8ba1ac92 |
fix(queue): show messages a booting run's first turn takes as sent (#20822)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Task chat lets a person send follow-up messages to a running agent, and steer them into the live turn > - A sandboxed run (for example on Daytona) can take minutes to boot before its provider session exists > - A message sent during that boot cannot be steered, so it waits in the steering pane > - The runner then puts the message into its first turn, and the server removes it from the queue, but the view has no signal for this, so the message stays in the steering pane as "queued" > - That stale local entry also hides the queue id, so Steer and Discard fail for every later message > - This pull request tells the view which messages the first turn takes, and drops local queue entries whose run has settled > - The benefit is that a message sent while the sandbox boots shows as sent at once, and later messages keep working Steer and Discard ## Linked Issues or Issue Description **What happened?** A person sent a follow-up while the agent's sandbox was booting. The message stayed in the steering pane with Steer unavailable. After the agent started, the message stayed there. The agent had received it in its first turn. Later, Discard on that message failed with "Couldn't discard. Message is still queued.", and a new message also stayed in the steering pane. **Steps to reproduce** 1. Assign a task to a Paperclip Runner agent that runs in a Daytona sandbox. 2. While the sandbox boots, send a follow-up message. 3. Wait for the agent's first response. The message is still in the steering pane. 4. After the run hands off to a successor run, send another message and try to discard the first one. **Expected behavior** A message sent while the run is starting shows as a sent message. Later messages keep working Steer and Discard. **Actual behavior** The first message stays in the steering pane. Discard fails, and the new message cannot be steered until the page reloads. ## What Changed - `firstTurnQueuedCommentIds` (server, `first-turn-queued-input.ts`) returns the messages in a deferred queue that the target run's first turn already sealed, or will take while it is still starting. It uses the same eligibility rule as `claimFirstTurnQueuedInput`. It reads only lean columns, so the queue poll does not transfer run snapshots or large wake payloads. - The queue read path and the queue mutation path both put these ids in a new optional `firstTurnCommentIds` field on `IssueQueuedCommentQueue`. - The task view puts each of those messages in the conversation as input of that run, at its submission time, and not in the steering pane. - The task view no longer treats a saved message as a pending local queue entry after its target run is no longer live. A request that is still in flight keeps its original target. ## Verification - Red-green server test: `issue-queued-comments-routes.test.ts` "reports the messages a preparing/native run's first turn takes, before and after it seals them" and "reports no first-turn messages once the run's turn already started". Before the fix, `firstTurnCommentIds` was `undefined`. - Red-green UI tests: `IssueDetail.test.tsx` "shows a message the booting run's first turn takes as sent, not queued" (still starting, and sealed) and "keeps queue controls for a new message after the run that took an earlier one hands off". Before the fix, the queue held `['new-message', 'boot-message']` and the queue id was withheld. - Reproduced on staging first: the run's context held a first-turn receipt for the stuck message, and the server queue had no entries, while the view still showed it as queued. - `npx vitest run` for `issue-queued-comments-routes`, `queued-comment-consumption`, `queued-comment-postgres`, `queued-comment-use-cases`: 172 passed. The existing queue-poll byte budget tests still pass. - `npx vitest run src/pages/IssueDetail.test.tsx src/components/task-chat src/lib/issue-queued-comment-queue src/lib/optimistic-issue-comments`: 1288 passed. - `tsc --noEmit` for shared, server, and ui: no new errors. - `pnpm verify` exits 2 because the change touches `packages/shared`. Required CI supplies the broad coverage. ## Risks - Low risk. The new field is optional and only for display. Delivery still belongs to the first-turn claim and the deferred wake. - If a starting run fails before its first turn, the field goes away and the message shows in the queue again for the next run. This is correct, because the deferred wake still holds it. - The queue read adds one lean run read and one comment read only while a deferred queue targets a running run. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file edits, browser). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Fable 5.2
|
a4b872477e |
fix(chat): do not wake the assignee for a decision its own run recorded (#20842)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A task can be bound to a chat conversation (Slack, Telegram). Confirmation cards are published there > - The assignee's running turn can record the user's chat reply as the decision on a card (resolve-from-comment). That run already owns the decision, and the in-app route does not wake the agent again > - The chat continuation worker (`interaction_wakeup`) still woke the assignee for that decision > - The wake waited behind the running turn. When it was promoted, the reply was already delivered, so its context kept `source: chat:<provider>` but no chat message > - Reviewed-chat attestation then refused the run (`reviewed_chat_execution_binding_not_authorized`, `context_mismatch`). The refusal is correct; the wake should not exist > - This pull request makes the worker settle that action without a wake > - The benefit is no failed setup run on the task, and no change to any permission check ## Linked Issues or Issue Description Refs #20809 **What happened?** On staging, an automation-triggered run failed in about 5 s with `setup_failed` and error `reviewed_chat_execution_binding_not_authorized` (denial reason `context_mismatch`). The task was bound to a Slack thread and was `in_review`. The assignee's previous run had recorded the user's Slack reply as acceptance of a plan confirmation and kept working. The chat continuation worker then queued a second wake for the same decision. That wake carried no undelivered chat message, so attestation found zero admitted comment ids. **Expected behavior** No second wake. The responding run already acts on the decision, as in the in-app path. **Steps to reproduce** 1. Bind a task to a Slack conversation and publish a `request_confirmation` card with `continuationPolicy: wake_assignee`. 2. While the assignee's run is active, the user replies in Slack. The run calls `POST /api/issues/:id/interactions/:interactionId/resolve-from-comment`. 3. The worker processes the `interaction_wakeup` chat action and wakes the assignee. The wake is deferred behind the active run and loses the delivered reply. With the task `in_review`, the next run fails attestation. ## What Changed - `server/src/services/chat-channels/interactions/continuations.ts`: when the current interaction was resolved by the assignee's own run from a comment (`resolvedByAgentId` is the assignee, `resolvedByRunId` is set, no `resolvedByUserId`, and `result.commentId` is set; only resolve-from-comment writes that), the worker marks the action `processed` with `interaction_wakeup_owned_by_responding_run` and creates no wake. Actions saved before the deploy take the same path. - `server/src/__tests__/chat-interaction-publications.test.ts`: real resolve-from-comment flow (no wake) and the board-accept control (one wake). - Reviewed-chat attestation and every authorization check are unchanged. ## Verification - Red on master: the new test fails, `wakeup` called 1 time. - Green: `server/src/__tests__/chat-interaction-publications.test.ts` 32/32 pass. - `pnpm --filter @paperclipai/server typecheck` passes. ## Risks - Low. Only a self-resolved, comment-recorded decision skips the wake. A user decision through a chat button or the board still wakes the assignee. - Not covered here: a user decision made through a chat button while another run of the task is active can still be deferred and lose its chat comment id. That is a separate case and is not changed by this PR. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude (Anthropic), model ID `claude-opus-5-5`, used through Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
4529dfb77f |
fix(runner): keep Codex planning read-only on Daytona without network namespaces (#20820)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A planning task runs a native Codex agent in read-only planning mode
before the user approves a Plan
> - On a Daytona sandbox, every Codex planning run failed before its
first turn: `error creating thread: failed to load AGENTS.md
instructions for environment 'local': fs sandbox helper failed ...
bwrap: loopback: Failed RTM_NEWADDR: Operation not permitted`
> - Codex 0.160.0 reads files through a filesystem helper when the
permission profile restricts reads. That helper always runs bubblewrap
with `--unshare-net`. Daytona has no `CAP_NET_ADMIN`, so bubblewrap
cannot bring up loopback in the new network namespace
> - The planning profile must stay read-only. Codex's plan collaboration
mode is only a prompt, so the external-sandbox profile (#20808 tried it,
then reverted it) would let planning runs write files
> - This pull request keeps the read-only profile, but inside an
external sandbox it gives the profile full-disk read, no writes and
network. Codex then reads files directly and runs commands without a new
network namespace, and it still refuses writes
> - The benefit is that Codex planning works on Daytona and stays
read-only
## Linked Issues or Issue Description
Refs #20808
**What happened?**
A Codex planning task on a Daytona execution environment failed at
startup. The run ended with `provider_session_open_rejected` and the
Codex message `bwrap: loopback: Failed RTM_NEWADDR: Operation not
permitted`. Standard and ask runs on the same sandbox worked.
**Expected behavior**
A planning run on Daytona starts, writes its Plan and asks for approval.
Codex must still refuse file writes and mutating commands during
planning.
**Steps to reproduce**
1. Configure a native Codex agent with a Daytona execution environment.
2. Create a task in planning mode.
3. The first run fails before its first turn with the bubblewrap
loopback error.
## What Changed
- `createIsolatedCodexAppServerArgs`: when
`PAPERCLIP_RUNNER_EXTERNAL_SANDBOX=1`, the
`paperclip-runner-workspace-read-only` profile is now `{":root"="read"}`
(plus the brokered `GH_CONFIG_DIR` write that planning already had). Its
network still follows `PAPERCLIP_RUNNER_NETWORK_ACCESS`. It has no deny
entries, so Codex 0.160.0 treats it as full-disk read and skips its
filesystem helper, which always unshares the network. Local and SSH
planning keep the narrower profile unchanged.
- The Codex session context now records `rootAccess` `read` (planning)
or `write` (default) in an external sandbox, instead of always `none`.
- The Runner still sends `permissions:
"paperclip-runner-workspace-read-only"` and the plan collaboration mode
on `thread/start` and `turn/start` for planning. A new Rust test pins
that for an externally sandboxed read-only launch (no `sandbox:
danger-full-access`, no `externalSandbox` `sandboxPolicy`).
- A new Rust test proves that a launch saved with the previous planning
profile rules has the same stable launch identity as a launch with the
new rules, so saved sessions carry forward.
- New manual Product E2E suite `codex-planning-read-only`
(`runner-codex` on local and Daytona). The planning turn runs one shell
write, writes a Plan whose first line records the outcome, asks for
approval, and finishes after approval. A grader reads the run events and
fails if no write attempt was seen or if any write succeeded.
- `docs/codex-driver.md` describes the external-sandbox planning
profile, why it avoids the helper, and its read scope.
## Verification
- Red on Daytona, master `123ee31552`: [run
43029207500](https://github.com/paperclipai/paperclip/actions/runs/43029207500),
cell `core-compatibility.runner-codex.daytona.plan-revise-accept` failed
twice with `error creating thread: failed to load AGENTS.md instructions
for environment 'local': fs sandbox helper failed with status exit
status: 1: bwrap: loopback: Failed RTM_NEWADDR: Operation not
permitted`.
- Planning turn completes on Daytona: [run
43031462803](https://github.com/paperclipai/paperclip/actions/runs/43031462803),
cell `core-compatibility.runner-codex.daytona.plan-revise-accept`. The
planning run succeeded, saved a two-step Plan with the draft marker and
opened a confirmation for that revision (all API checks passed). The
cell then failed a later browser check: the Plan preview rendered as the
`task-chat-plan-preview-fallback` card, which the harness does not
accept. That is a separate UI placement gap.
- Write attempt on Daytona: [run
43038547892](https://github.com/paperclipai/paperclip/actions/runs/43038547892),
cell `codex-planning-read-only.runner-codex.daytona.plan-write-refused`
on `f30aeb3e72`. The planning run succeeded and ran the shell write
`printf probe > plan-write-probe-....txt || echo WRITE_REFUSED`. It
ended `failed` with exit 1 and no file, because bubblewrap could not
start the command (`bwrap: setting up uid map: Permission denied`). The
write was refused fail-closed by the sandbox launcher, not by a
read-only filesystem error. The stricter grader in
|
||
|
github-actions[bot]andlockfile-bot
|
1d67e309e0 |
chore(lockfile): refresh pnpm lockfiles (#20845)
## Thinking Path > - Paperclip uses a dependency lock for reproducible builds. > - Dependency changes on master need matching lock entries. > - Paid eval reports also pin a reviewed offline correction. > - Reports must install before paid model work starts. > - This automated PR keeps the lock and report proof together. ## Linked Issues or Issue Description **What happened?** Dependency manifests on master changed after the last committed lock refresh. **Expected behavior** Frozen installs must match those manifests. Eval reports must retain matching offline lock proof. **Steps to reproduce** Run a frozen install before and after this lock refresh. ## What Changed - Refresh the dependency lock from the manifests on master. - Refresh the sandbox provider lockfiles from their manifests on master. - Include the matching offline report correction and hashes. - Resync the SDK public source proof's lock pins from the refreshed lock. ## Verification - Pinned pnpm resolution completed with lifecycle scripts disabled. - The report proof refresh accepted the reviewed committed baseline. - The workflow rejected changes outside the lock and report proof files. ## Risks The lockfile bot merges this PR without CI or review when every added package version is at least 3 days old on npm. Otherwise the PR waits for a maintainer, and the daily refresh tries again. The bot merges only the exact commit this workflow generated from manifests already reviewed on master. Unknown report lock baselines remain blocked. This update changes no permissions or paid enable gates. ## Model Used None — generated by pnpm and the deterministic report-lock updater. ## Checklist - [x] Thinking path and problem description included - [x] Generated lock and matching report proof included - [x] Release-age gate decides between bot merge and maintainer review Co-authored-by: lockfile-bot <lockfile-bot@users.noreply.github.com> |
||
|
Nicky LeachandClaude Opus 5.5
|
ff8ec41e6a |
feat(runner): upgrade ACPX to 0.22.0 and move patched behavior into Paperclip (#20770)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Paperclip Runner and the shared `adapter-utils` engine run ACP agents through the embedded ACPX library > - Paperclip pins ACPX 0.13.1 and keeps a large patch (`patches/acpx@0.13.1.patch`) on it > - ACPX 0.22.0 adds host hooks (`onAcpMessage`, `assertActive`, `processLifecycle`, prompt-response `_meta`, `spawnAgent`-friendly launch) that make much of that patch unnecessary > - A large patch is expensive to port and hides behavior that belongs to Paperclip > - This pull request upgrades to ACPX 0.22.0 and moves most patched behavior into Runner and `adapter-utils` > - The benefit is a patch that is 484 lines instead of 1,334, with the moved behavior covered by Paperclip tests ## Linked Issues or Issue Description Refs #20675, #14430, #15075, #17942, #9980 **Proposed enhancement** Upgrade the embedded ACPX library from 0.13.1 to 0.22.0. Keep only the patch behavior that stock 0.22 cannot express. **Motivation** The 0.13.1 patch held behavior that Paperclip owns, for example usage receipts, Copilot policy, model checks and the agent environment. Each ACPX upgrade had to port all of it. ACPX 0.22 gives hooks that let Paperclip do this work itself. **Current behavior** Runner and `adapter-utils` depend on 1,334 lines of patched ACPX 0.13.1 code. **Expected behavior** Runner and `adapter-utils` use ACPX 0.22.0. The patch keeps only: extension methods, Cursor subagent routing, event `_meta` and the message boundary chunk, the protocol guard hook, `spawnAgent`, and `requestExtension`. ## What Changed - ACPX 0.13.1 → 0.22.0 in Runner, `adapter-utils`, the server, the Daytona runner images and the qualified profiles (`acpxVersion`). `@agentclientprotocol/sdk` moves to 1.9.0. - New `patches/acpx@0.22.0.patch` (484 lines). The 0.12.0 and 0.13.1 patches are removed. - Delivery receipts: Runner confirms a permission reply when `onAcpMessage` sees the exact outbound reply. Question and extension replies count as delivered when Runner hands them back to ACPX. New `interaction-delivery.ts`. - Copilot policy: the existing policy function runs from `onAcpMessage`. A violation stops the agent, cancels later permission requests and fails the next prompt through `assertActive`. - Model selection: stock 0.22 validation, plus a sidecar gate (`createAcpxEffectiveModelGate`) that refuses a prompt if the agent reports another model. Grok binds its model when the session opens. Antigravity adds its concrete model to the model list ACPX reads. - Typed session failures: Runner and `adapter-utils` read them from the turn result `_meta` and fail the turn. They advertise the capability themselves. - Pi and Cursor usage receipts: a collector (`prompt-receipts.ts`) binds them to the turn's own `session/prompt` reply. - Goal capability: Runner reads it from the `initialize` reply. It keeps it under `paperclip.goalCapability` in the saved agent capabilities `_meta` and restores it when a saved session loads. - Agent environment: Runner and `adapter-utils` set the agent environment in `spawnAgent`. ACP terminals keep their current environment. `adapter-utils` blocks inheritance of `XAI_API_KEY` and refuses to start if the server has an `ACPX_AUTH_*` variable. - `adapter-utils` spawn hooks (`spawnCwd`, stderr, `onAgentSpawn`): a new `stockAcpRuntimeOptions` maps them to stock `spawnAgent` and `processLifecycle`. - Windows: ACPX 0.22 rejects raw command strings on Windows. `adapter-utils` now passes an argv array there. - Grok rules: a Runner guard (`grok-session-rules.ts`) writes `session/new` `_meta.rules`. - Turn-dispatch journal (#20675): ACPX 0.22 ignores errors from `onAcpMessage`. The adapter keeps a failed `dispatched` write and throws it from the turn's `assertActive`. ACPX calls `assertActive` before the prompt write, so the write still stops. The real-ACPX test now checks this route. `docs/durable-recovery.md` is updated. - Plan events: the sidecar reads 0.22 `status` events tagged `plan`. - CI: the backup-checkpoint lane now installs through the shared `setup-ci-dependencies` action, like PR CI, so it resolves a stale `pnpm-lock.yaml` in place. It runs on any root `package.json` change, and pull requests never commit the lockfile, so its own frozen install failed on dependency changes. - Tests that checked removed patch internals are removed or rewritten: the response-delivery, Cursor-usage and Pi-receipt package contracts, and the goal-persistence packaging test. New unit tests cover the moved behavior. ## Verification Local runs on macOS (darwin-arm64), Node 26, on the rebased head: - `pnpm --filter @paperclipai/paperclip-runner exec vitest run src/drivers/acpx src/cli`: 2,000 passed. - `tsc -p packages/paperclip-runner/tsconfig.json` and `tsconfig.test.json`: no errors, except the existing `@paperclipai/paperclip-eval-kernel` build gap in `workflow-harness.ts`. - Runner package contracts (`node --test test/acpx-*-package-contract.test.mjs test/acpx-sidecar-contract.test.mjs`), `scripts/build-inputs.test.mjs`, `scripts/contracts/contract-fingerprints.test.mjs`, `scripts/acpx-patch-packaging.test.mjs`: all pass. `node scripts/generate-acpx-sidecar-contract.mjs --check` passes. - `cargo test -p paperclip-runner-core --test acpx_park_preparation --test acpx_park_readiness --test acpx_warm_attachment --test native_provider_backend`: 86 passed. `cargo fmt --check` passes. - `adapter-utils`: `tsc` clean. `vitest run src/acpx-engine src/mcp-isolation.integration.test.ts`: 457 passed (one Gemini flag test failed once under load and passes when run alone). - Adapters: `claude-local` and `codex-local` `acp.quota.test.ts` (23 and 39 passed), `grok-local` (149 passed). - Server: `scripts/prepare-runner-vendor.test.mjs`, `native-session-executor.test.ts`, `quarantined-task-retry.test.ts` (1,416 passed). - `cargo test -p paperclip-runner-core --lib -- previous_acpx_release acpx_upgrade unrestorable_saved_sessions`: 3 passed. `node --test test/acpx-session-carry-forward-package-contract.test.mjs`: passes. - `patch -p1 --forward` on a clean `acpx-0.22.0.tgz` gives output identical to the port. - Under full-suite load, some Runner live and recovery tests time out. Each one passes when run alone. Full regression is left to CI. Not run: live provider runs (they need provider credentials), the opt-in Kimi live peer test, the Gemini filesystem smoke test, and Windows. ## Risks - Previous-release story: sessions saved by the previous release (ACPX 0.13.1) carry forward. ACPX 0.22 loads the session records 0.13.1 wrote. Runner moves a saved descriptor that names 0.13.1 (`CARRIED_ACPX_RELEASES`) to 0.22.0, and the sidecar keeps 0.13.1 as the ACPX value in the session identity, so the saved session key and identity record still match. The existing launch-profile rebind still decides which sessions resume: a settled session, or the same run's interrupted turn through lost-turn recovery. A session saved by any other ACPX release is still refused as `provider_state_incompatible`. `rules/acpx-profile-identity.mdc` and `docs/durable-recovery.md` now describe this. - Proof: `acpx-session-carry-forward-package-contract.test.mjs` resumes a session record written by the real ACPX 0.13.1 with its Paperclip patch. 0.22 loads the same provider session (no `session/new`), runs a turn and keeps the earlier turn. Rust tests cover a settled 0.13.1 session that the next run attaches to, and a turn interrupted by the upgrade. A recovery-identity test checks that the session key and identity record match across the versions. - `pnpm-lock.yaml` is not committed. CI resolves it for each run, and the lockfile refresh workflow updates it after merge. - Behavior change on the server: `adapter-utils` refuses to create an ACPX runtime if the server environment has an `ACPX_AUTH_*` variable. Before, ACPX ignored such variables. Stock 0.22 would use them to authenticate agents with server credentials. - Delivery confirmation is now Runner-owned. A permission reply is confirmed by the exact outbound message. Question and extension replies are confirmed when Runner hands them to ACPX, not when the pipe accepts them. - Users can select only models that the agent advertises. Stock 0.22 rejects other model IDs (Claude is exempt). - ACP terminals keep the sidecar or server environment, as before. This existing exposure is not changed by this pull request. - The `adapter-utils` Gemini version probe now runs with the server environment plus the run variables, not the run variables alone. - The pack contract digest and profile `acpxVersion` change, so Daytona needs a new runner image. - Grok launch flags (`--model`, `--reasoning-effort`) were checked on 1.0.13 but session open needs a login, so Grok keeps the existing `_meta` path. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file editing, subagents). Extended reasoning was used. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
34f2bb78ee |
feat(connections): connect Sentry with a long-lived user auth token (#20823)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents use app connections to call provider tools, such as the
Sentry MCP server
> - The Sentry connection has only one method: browser sign-in through
the hosted MCP OAuth server
> - That grant expires. An unattended triage agent loses Sentry access
until a person signs in again
> - During that time, a scheduled triage routine completes each run but
reads no Sentry data
> - This pull request adds a second Sentry method that uses a user auth
token
> - The benefit is a Sentry connection that stays valid until a person
revokes the token
## Linked Issues or Issue Description
**Subsystem affected**
App connections: the Sentry entry in the connection catalog.
**Problem or motivation**
A Sentry connection that was created with browser sign-in on 2026-10-11
changed to "OAuth authorization expired. Reconnect this app to
continue." at 04:01 UTC on 2026-11-10. The agent that used it had no
Sentry tools after that time. Each hourly triage run reported the
blocker but could not continue. Unattended agents need a Sentry
connection that does not need a person to sign in again each month.
**Proposed solution**
The hosted Sentry MCP server accepts a Sentry user auth token in the
`Authorization: Sentry-Bearer <token>` header. It reserves `Bearer` for
its own OAuth tokens. Paperclip API-key methods already support a custom
header prefix. Add a Sentry API-key method that uses this header.
**Alternatives considered**
Reconnect with browser sign-in again. That restores access only until
the next grant expiry.
## What Changed
- Add the `mcp-user-token` method ("Use a user auth token") to Sentry in
`scripts/ingest-app-definitions.mjs`. It uses the same server URL and
sends the token as `Authorization: Sentry-Bearer <token>`.
- Regenerate `packages/shared/src/app-definitions/sentry.json`. I did
not include unrelated output that the content corpus drift caused for
other providers.
- Add a permission review entry for the new method in
`doc/connections/tool-method-permission-reviews.json`.
- Add a test for the new method and its header prefix.
- Document the token method, its required scopes, and its ownership risk
in `doc/connections/SENTRY.md`.
## Verification
- `pnpm exec vitest run packages/shared/src/app-definitions.test.ts`: 40
passed. The new assertion fails on `master` because Sentry has only one
method there.
- `pnpm exec vitest run src/__tests__/tool-access-service.test.ts -t
Sentry` (server): 2 passed.
- `pnpm exec vitest run ui/src/pages/apps/app-connect-policy.test.ts
server/src/__tests__/satisfied-connection-intents.test.ts`: 64 passed.
- `pnpm verify` returned exit 2 (a doc is a global CI input). Required
CI supplies the broad coverage.
- Live probe of `https://mcp.sentry.dev/mcp` with an `initialize`
request: `Authorization: Sentry-Bearer <invalid token>` starts an MCP
session (the token is checked on tool calls). `Authorization: Bearer
<same token>` returns `401 invalid_token`. This confirms the server
accepts the header scheme.
- Not done yet: a real agent tool call with a real token after
deployment.
## Risks
- Low risk. This adds a method. The existing OAuth method and saved
connections do not change.
- A user auth token acts as its creator and lasts until revoked. The
method guidance and docs say this. Rotate the token when that person
leaves.
- If Sentry changes the `Sentry-Bearer` scheme, connections that use
this method fail on tool calls. The OAuth method still works.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, browser, web search).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
Nicky LeachandClaude Opus 5.5
|
6ce953ca52 |
fix(runner-e2e): resync the SDK lockfile pins to master's current lock (#20834)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Runner support suite proves the plugin SDK public types from pins in `tests/runner-e2e/source-sdk-public-inputs.json` > - Two of those pins follow the root lockfile: the `pnpm-lock.yaml` blob, and `lockSha256`, which must equal the `outputSha256` in `tests/runner-e2e/report-lock.json` > - The lockfile refresh #20824 changed the lockfile and `report-lock.json`. The registry kept the old values > - #20623 makes the refresh resync these pins, but its registry values come from the refresh before #20824 > - The refresh resyncs only when it commits a lockfile change. The lockfile is current, so the refresh after #20623 did nothing, and the pins stay stale > - This pull request regenerates the two pins with the existing `--write` command > - The benefit is that `ci / Runner support suite` passes again on master and on new pull requests ## Linked Issues or Issue Description Refs #20824. Refs #20623. **What happened?** `ci / Runner support suite` fails on master at `e2cf9a670f` and on pull requests that merge master. `source-sdk-public-type.test.ts` and `source-sdk-evidence.test.ts` fail with "Reviewed SDK reference policy does not match public proof". `source-sdk-public-inputs.test.mjs` reports `pnpm-lock.yaml (changed)`. The refresh run after #20623 merged (run 43043576648) logged "Lockfiles and reporting-lock proof unchanged, nothing to do". **Expected behavior** The registry pins the committed lockfile, and `lockSha256` equals the `outputSha256` in `report-lock.json`. **Steps to reproduce** 1. Check out master at `e2cf9a670f`. 2. Run `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/source-sdk-public-type.test.ts tests/runner-e2e/source-sdk-evidence.test.ts tests/runner-e2e/source-sdk-public-inputs.test.mjs`. 3. See the "Reviewed SDK reference policy does not match public proof" failure. ## What Changed - `tests/runner-e2e/source-sdk-public-inputs.json`: regenerated with `node tests/runner-e2e/source-sdk-public-inputs.mjs $(git rev-parse HEAD) --write`. Only two values change: - `inputs["pnpm-lock.yaml"]`: `6d825f97…` → `d24edac1…`, the committed lockfile blob. - `lockSha256`: `b3208306…` → `17625672…`, the `outputSha256` in `report-lock.json`. - No other pin changes. The lockfile and `report-lock.json` do not change. ## Verification - Before the change, the three SDK proof test files fail with the policy mismatch. After the change, they pass. - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/source-sdk-public-type.test.ts tests/runner-e2e/source-sdk-evidence.test.ts tests/runner-e2e/source-sdk-public-inputs.test.mjs tests/runner-e2e/prepare-report-lock.test.ts`: 4 files, 68 tests pass. - `node tests/runner-e2e/source-sdk-public-inputs.mjs $(git rev-parse HEAD) --check` reports `changed: false`. - Local run on macOS (darwin-arm64). Full regression is left to CI. ## Risks - Low risk. The change is test data only, and it uses the values that the registry command derives from the committed lockfile and the reviewed `report-lock.json`. - The same two values are in #20770. Whichever pull request merges first, the other one merges cleanly. - The refresh still resyncs these pins only when it commits a lockfile change. If the pins become stale with no lockfile change, the refresh does not repair them, and the support suite shows the failure and the fix command. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file editing). Extended reasoning was used. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
693743cd59 |
fix(ui): refresh dependent tasks when a blocker changes (#20836)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Tasks can depend on other tasks. A task detail shows its blockers in the properties panel and in a "Still blocked by" notice. > - The task detail response embeds a summary of each blocker, including its status. > - Live updates refresh only the changed task's own cached detail. They do not refresh other cached tasks that embed it. > - So when a user marks a blocker done, an open dependent task keeps the old blocker status. The "Blocked by" chip shows "blocked" while the Subtasks chip shows "done". The page looks stuck until a reload or until a run starts. > - This pull request refreshes every cached task detail that embeds the changed task in `blockedBy` or `blocks`. > - The benefit is that dependent tasks show the current blocker status immediately. ## Linked Issues or Issue Description **What happened?** A user marked a blocker task done by hand. The dependent task page still showed the blocker with a "blocked" icon and a "Still blocked by" notice. The same blocker showed as "done" in the Subtasks row of the same panel. The server had already resolved the dependency and woke the dependent task. Only the open page was stale. **Steps to reproduce** 1. Open task A, which is blocked by its subtask B. 2. In another view, mark B done. 3. Go back to A without a reload. "Blocked by" still shows B as blocked, and the notice says "Still blocked by B". "Subtasks" shows B as done. **Expected behavior** The "Blocked by" chip and the notice update when B changes, the same as the Subtasks row. ## What Changed - `LiveUpdatesProvider`: when activity changes a task's status, retry or recovery fields, invalidate every cached `issues.detail` query whose `blockedBy` or `blocks` summaries contain that task. The match uses the UUID or the identifier, because authenticated events can omit the identifier. - Add a regression test for this in `LiveUpdatesProvider.test.ts`. ## Verification - Red: the new test failed before the fix. The dependent detail was not invalidated. - Green: `pnpm vitest run src/context/` in `ui` passes (311 tests). - `pnpm verify` passes (549 selected test files). - `tsc -p ui --noEmit` shows no new errors. Existing errors in `ui/storybook/support/feature-map` are unrelated. - Staging data check: the server returned the blocker as `done` in the dependent's `blockedBy`. The stale display came only from the client cache. ## Risks Low risk. The predicate scans only cached task details. It refetches only details that embed the changed task. Refetches use the existing invalidation batcher. The list of actions excludes comments and document edits, so busy tasks do not cause extra refetches of their dependents. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
e2cf9a670f |
fix(runner-e2e): resync SDK public lock pins and make the lockfile bot keep them current (#20623)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Runner support suite proves the plugin SDK public types from pinned sources in `tests/runner-e2e/source-sdk-public-inputs.json` > - That registry pins the committed `pnpm-lock.yaml` blob and a `lockSha256` that must equal the `outputSha256` in `tests/runner-e2e/report-lock.json` > - The lockfile bot refresh #20615 changed the lock and `report-lock.json`, but the bot does not write the SDK registry > - The bot merges without CI, so the drift guard from #20579 could not block it, and every PR now fails `ci / Runner support suite` > - The guard also checked only the source blobs, so it did not name the `lockSha256` drift, and the sync script could not update `lockSha256` > - This pull request regenerates the registry, derives `lockSha256` from the reviewed policy, and makes the bot resync and check the registry in its own commit > - The benefit is that master CI is green again, and the next lockfile refresh cannot leave the SDK proof stale ## Linked Issues or Issue Description Refs #20615. Refs #20579. Refs #20550. Refs #20613. Overlaps #20618, which refreshes only the registry data. This PR has the same data change and also closes the gap that let the drift land. **What happened?** After #20615 merged, `ci / Runner support suite` failed on master and on every PR (for example runs 42877750315 and 42877750957, and PR #20613). `source-sdk-public-type.test.ts` and `source-sdk-evidence.test.ts` failed with "Reviewed SDK reference policy does not match public proof". `source-sdk-public-inputs.test.mjs` failed with "pnpm-lock.yaml (changed)". **Expected behavior** A lockfile refresh keeps the SDK public proof registry in step with the lock and the reviewed report-lock policy. If the registry drifts, the failure names the stale pin and the fix command. **Steps to reproduce** 1. Check out master at `3bf7af6c77`. 2. Run `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts source-sdk`. 3. See the "Reviewed SDK reference policy does not match public proof" failure. ## What Changed - Root cause: #20615 changed `report-lock.json` from output `2360d5e9…` to `b3208306…` and changed the `pnpm-lock.yaml` blob from `851e65af…` to `6d825f97…`. The registry kept the old values. `refresh-lockfile.yml` commits only the lockfiles and the report-lock proof, and it merges without CI. - `source-sdk-public-inputs.json`: regenerated with `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --write`. Only `inputs["pnpm-lock.yaml"]` and `lockSha256` change. The compiler, parser and Zod pins do not change. - `source-sdk-public-inputs.mjs`: the sync now copies `lockSha256` from the committed `report-lock.json` `outputSha256`. It refuses when the committed lock is not that policy's reviewed input or output. This is the same rule that `prepareReviewedSdkReference` applies. Only the lockfile bot can write that policy, so a contributor cannot choose `lockSha256`. - `staleSdkPublicInputs()`: also reports a `lockSha256` that differs from the committed report-lock output. The guard test then names both stale pins and the `--write` command. - `source-sdk-reviewed-fixture.ts`: the "Reviewed SDK reference policy does not match public proof" error now names the two values and the fix command. The check itself is unchanged. - `refresh-lockfile.yml`: after the lock commit, the bot runs the SDK sync with `--write`, amends the commit, and runs `--check` before it pushes. `head_sha` is still read after the amend, so the bot merges the exact commit it checked. - Tests: new sync tests for a lockfile refresh and for an unreviewed lock, and a workflow-order test in `lockfile-refresh-merge.test.mjs`. - Docs: `doc/RELEASE-AUTOMATION-SETUP.md` §8.1 and `tests/runner-e2e/OPERATIONS.md` describe the resync. ## Verification - Reproduced on master: `source-sdk-public-type.test.ts` and `source-sdk-evidence.test.ts` failed with the policy mismatch. - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts source-sdk prepare-report-lock sdk-canary workflow-security`: 9 files, 135 tests pass. - Before the registry change, the extended guard reported `pnpm-lock.yaml (changed), tests/runner-e2e/report-lock.json outputSha256 (lockSha256 changed)` with the fix command. - `node tests/runner-e2e/source-sdk-public-inputs.mjs $(git rev-parse HEAD) --check` reports `changed: false`. - `node --test .github/scripts/tests/lockfile-refresh-merge.test.mjs .github/scripts/tests/lockfile-refresh-workflows.test.mjs .github/scripts/tests/lockfile-trust.test.mjs`: 39 tests pass. - `pnpm test:e2e:runner:unit` on a host with load near 28: 6 files timed out, and none of them are the SDK proof files. All 15 of those files pass when run alone. CI runs the full suite. ## Risks - The bot step now runs one more repository script. If that script refuses (for example, an unreviewed committed lock), the refresh PR is not pushed. Before this change, the bot merged a refresh that made master red. - `lockSha256` now comes from the reviewed policy, not from the old registry value. The proof check is unchanged. It still needs the lock bytes to hash to `lockSha256`. - If #20618 merges first, this branch rebases cleanly, because the registry change is identical. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edit) and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude
|
fd1460bb24 |
fix(native): continue a governed-wait follow-up in a fresh session when its saved provider state is gone (#20809)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness (`tests/restart-chaos/`) runs real agent tasks and breaks the control plane on purpose, so failures show up before staging users see them > - A census of 48 hours of staging failures found runs whose result save-back failed because the sandbox disk was full (class F). The harness never filled a sandbox disk, so it missed that class > - #20758 made a full-disk save end recovery at once with cause `sandbox_disk_full` and a Retry. Nothing in chaos exercised that path end to end > - A real full disk cannot be made on demand. The owner chose a test-only hook that makes the save report a full disk, not a small disk > - This pull request adds the hook at the save's sandbox boundary and a DF1 fault that arms it, clears it, and grades the result > - The benefit is that every wave can now check the disk-full rest, the Retry offer, and that Retry works after space is freed ## Linked Issues or Issue Description Refs #20758 (product behaviour under test), #20678 (finalization hands a full-disk copy-back to recovery) **What is the problem?** The chaos harness had no fault for a full sandbox disk during result save-back. Staging saw this failure (census class F in `doc/plans/2026-11-11-staging-failures-vs-chaos-coverage.md`), and chaos could not reproduce it. **What is the expected behaviour?** A wave can schedule DF1. While DF1 is armed, the task's final workspace saves fail as a full disk. When DF1 is cleared, saves succeed again. The task must rest with `sandbox_disk_full` and a Retry, and that Retry must finish the task. **Why does it matter?** Without a fault, a regression in the disk-full path (no Retry, a wrong cause, or a Retry that does not work) reaches staging users before chaos sees it. ## What Changed - `server/src/services/sandbox-disk-full-fault.ts` (new, test-only): `sandboxDiskFullFaultTarget` wraps the target of a final save only when the harness armed that save. The capture commands then exit 1 with the exact message the in-sandbox script writes on ENOSPC, and the provider download fails with `No space left on device`. The gate follows `@paperclipai/shared/crash-points`: `PAPERCLIP_DISK_FULL_FAULT_CONTROL` is read once at import and ignored under `NODE_ENV=production`. Disarmed, the hook is one comparison and returns the save's own target. - `server/src/services/remote-workspaces.ts`: the capture call in `saveRemoteWorkspace` passes its target through the hook. No other product change. `workspace-recovery.ts` is not touched. - `tests/restart-chaos/disk-full-faults.ts` (new): the DF1 kind, the control-file seam (atomic writes), the rest evidence (`activeRecoveryAction.evidence.systemRecovery.failure.cause.label`), the `disk_full_fault` T check, and the DF1 R3 rule (one expected Retry that must finish the task). - Harness wiring: `--faults DF1` (refused without `--env daytona`, with a storm, or with PV faults), no scheduled restart, arm at launch, clear at the first rest just before the Retry click, receipts in `<wave>/disk-full-faults.jsonl`, and ledger fields `diskFull: { faultedSaves, restedDiskFull, retryOffered, retryWorked }`. - Docs: a "Sandbox disk full (DF1)" section in `tests/restart-chaos/README.md`, and the census plan marks DF1 as built. ## Verification - `cd server && pnpm exec vitest run src/services/sandbox-disk-full-fault.test.ts`: the gate is off by default and under `NODE_ENV=production`, the save's own target comes back when disarmed, only a listed final save is faulted, the product capture turns it into the named disk-full failure, and clearing the file disarms the next save. - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts tests/restart-chaos/disk-full-faults.test.ts tests/restart-chaos/grader.test.ts tests/restart-chaos/scheduler.test.ts tests/restart-chaos/provider-faults.test.ts`: planning, the control file, and grading. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos` and the server `tsc --noEmit`: clean. - Live wave: a Daytona DF1 wave is running. Its result is added here before merge. ## Risks - Low risk for production. The hook reads its variable once at startup and is ignored under `NODE_ENV=production`. Without the variable it does one comparison and returns the same target object. - The hook fails every sandbox command of an armed capture, including the read-only fallback's inventory. A real full disk can let that inventory succeed and fail later at the download. The outcome is the same (`sandbox_disk_full`), but the fallback's inventory step is not exercised under DF1. - DF1 runs only with `--env daytona`. A local run has no sandbox workspace save, so the harness refuses DF1 there. ## Model Used - Claude Opus 5.5 (Anthropic, model ID `claude-opus-5-5`), in Claude Code with tool use (shell, file edits, GitHub CLI). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
5f38226290 |
fix(build): report compiler signals in server typecheck layers (#20603)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Its development checks help contributors verify changes.
> - The server typecheck runs the compiler once for each source layer.
> - A compiler child can stop on a signal before it reports type errors.
> - The wrapper returned failure without the signal or the affected
layer.
> - This pull request reports the signal, layer, command, and working
directory.
> - Contributors can identify an interrupted check from its output.
## Linked Issues or Issue Description
**What happened?**
The server typecheck returned exit 1 after a layer banner when a
compiler child stopped on a signal. The wrapper did not report that
signal. Related layer work: #18737, #18738, and #18739. A search of open
and closed PRs found no duplicate signal-reporting fix.
**Expected behavior**
Report the actual termination signal and affected layer. Keep compiler
output visible and return failure. Do not infer a memory failure from
SIGKILL alone.
**Steps to reproduce**
Run a controlled compiler child that sends SIGTERM or SIGKILL to itself.
Run it through typecheckLayers and the real runTsc subprocess path. The
added tests do this without memory pressure or timed waits.
**Paperclip version or commit**
Reproduced on
|
||
|
github-actions[bot]andlockfile-bot
|
a4e89288ec |
chore(lockfile): refresh pnpm lockfiles (#20824)
## Thinking Path > - Paperclip uses a dependency lock for reproducible builds. > - Dependency changes on master need matching lock entries. > - Paid eval reports also pin a reviewed offline correction. > - Reports must install before paid model work starts. > - This automated PR keeps the lock and report proof together. ## Linked Issues or Issue Description **What happened?** Dependency manifests on master changed after the last committed lock refresh. **Expected behavior** Frozen installs must match those manifests. Eval reports must retain matching offline lock proof. **Steps to reproduce** Run a frozen install before and after this lock refresh. ## What Changed - Refresh the dependency lock from the manifests on master. - Refresh the sandbox provider lockfiles from their manifests on master. - Include the matching offline report correction and hashes. ## Verification - Pinned pnpm resolution completed with lifecycle scripts disabled. - The report proof refresh accepted the reviewed committed baseline. - The workflow rejected changes outside the lock and report proof files. ## Risks The lockfile bot merges this PR without CI or review when every added package version is at least 3 days old on npm. Otherwise the PR waits for a maintainer, and the daily refresh tries again. The bot merges only the exact commit this workflow generated from manifests already reviewed on master. Unknown report lock baselines remain blocked. This update changes no permissions or paid enable gates. ## Model Used None — generated by pnpm and the deterministic report-lock updater. ## Checklist - [x] Thinking path and problem description included - [x] Generated lock and matching report proof included - [x] Release-age gate decides between bot merge and maintainer review Co-authored-by: lockfile-bot <lockfile-bot@users.noreply.github.com> |
||
|
DottaandPaperclip
|
81c9143fe4 |
fix: upgrade Better Auth to 1.7.16 with account migration (#18906)
## Thinking Path > - Paperclip manages AI agents and their work. > - The board uses Better Auth for credential accounts and sessions. > - The server pins Better Auth 1.7.2. > - The November security release fixes 12 issues in Better Auth. > - The new version stops writing the old required account issuer field. > - This change updates the packages and the database contract together. > - New users can sign up, and existing users can still sign in. ## Linked Issues or Issue Description **What happened?** The server uses Better Auth 1.7.2. The [security release](https://better-auth.com/blog/security-release-2026-11-03) fixes 12 issues in 1.7.16. Earlier update PRs #17909 and #17911 target 1.7.7, which is below this security release. **Expected behavior** Use the patched stable release. Keep new and existing credential accounts usable. **Steps to reproduce** Read the server dependency pin. Upgrade it without a schema change. Better Auth then rejects the required account issuer column. See the [upgrade guide](https://better-auth.com/docs/guides/1-7-upgrade-guide). ## What Changed - Pin Better Auth to 1.7.16. The repository refresh bot owns the committed Core lockfile. - Make the old issuer column nullable. Keep existing issuer values. - Replace the issuer index with a unique provider/account index. - Generate the migration and retain the five newest schema snapshots. - Test new signup, new-account sign-in, legacy issuer sign-in, duplicate rejection, and separate provider identities with real PostgreSQL. - Update the database schema test. Test the historical issuer backfill separately. Verify upgrades over existing rows and rollback on duplicate provider/account keys. - Document the duplicate check and migration order. - Add the approved two-line dependency resolution step before the backup job’s frozen install. Leave the frozen check and backup tests intact. ## Verification - Core head: `af70282c7e24335b8688c8f3cc88fdf65045632c`; parent `0f1481e4462100357a7caa496b875cf0bc500d95` includes master `84fc67317`. - Nine changed files. The only workflow edit is the approved two-line backup dependency resolution step; no managed lockfile edits. - Migration `0465_better_auth_account_identity.sql` is three SQL statements; snapshot changes only `public.account`, five newest snapshots retained. - DB typecheck and 23 focused tests passed on the parent. Both real PostgreSQL migration/signup suites were rerun November 11: all 4 tests passed. Application source and tests are unchanged in this workflow-only commit. - The parent’s sandbox-provider lane passed all 205 file-sync tests but reported an unhandled EPIPE during the symlink rejection case; its aggregate verify therefore failed. No unrelated source change was made. The new head reruns the full applicable CI. - Reproduced the old backup install failure locally. Resolving manifests followed by a frozen install passed locally; restored the bot-owned committed lock afterward. Hosted backup qualification remains required. - Current-head CI and review are running; no final-head green claim or merge yet. ## Risks Apply the migration before the new server starts. Check for duplicate provider/account pairs first. Duplicate rows stop index creation. Keep separate users separate when resolving duplicates. The migration keeps issuer data. Do not restore its NOT NULL constraint while new rows contain null. No deployment is included. ## Model Used OpenAI GPT-6 Codex, with reasoning and tool use. The runtime did not expose a more specific model ID or context window. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
aaf1337c7b |
fix(slack): require mentions only for channel roots (#20821)
## Thinking Path > - Paperclip helps people manage AI agents for work. > - Slack connects people to those agents through channels and threads. > - A mention can start a conversation without making the bot answer all channel traffic. > - The current setting also requires mentions on every thread reply. > - This interrupts an existing conversation and can cancel a queued reply. > - This change requires mentions on channel roots and allows replies without a mention. > - People can continue a thread without adding the bot name to each message. ## Linked Issues or Issue Description **What existing behavior does this improve?** The Slack connection setting that requires an at-mention. **Current behavior** When the setting is on, a channel thread reply must mention the bot. The delivery and execution checks reject a reply that has no mention. **Proposed behavior** Require a mention on a top-level channel message. Allow replies in existing threads without a mention. Keep the current DM access controls. **Reason and benefit** A person can continue an existing conversation without repeating the bot name. **Breaking changes** Existing connections with the setting on will accept thread replies without a mention. The saved setting, default value, and API field stay the same. ## What Changed - Rename the control to **Require at-mention in channels**. Explain the thread and DM rules in the help text and operator documentation. - Keep the mention policy in the Slack module. Use the thread root and message IDs to identify a reply. - Call that policy from durable admission, wake authorization, and queued run start. These shared paths must enforce the same setting when a delivery resumes. Other providers keep their current behavior. - Update PostgreSQL integration tests for thread replies, DMs, queued deliveries, and combined wake inputs. Add tests that reject incomplete reply identity. ## Verification - Reproduced rejected thread replies and cancelled queued reply runs before the fix. - Passed 16 focused PostgreSQL integration cases after the fix. - Passed 71 tests on revision `31b1adede2b92bad24b8fa21b8f6fffa9bd26d04` after the rebase. This includes the full durable wakeup suite, Slack policy tests, and eight selected Slack delivery cases. - Command: `pnpm exec vitest run server/src/services/chat-channels/slack/wake-authority.test.ts server/src/__tests__/durable-chat-wakeup.test.ts server/src/__tests__/chat-channels.integration.test.ts -t "at-mentions by default|thread replies without mentions|rechecks queued Slack|preserves running Slack tool authority|Slack channel mention policy|durable inbound chat scheduler receipts"`. - Strengthened the queued-root test to check the mention-rejection reason and content redaction. A temporary change that disabled mention enforcement caused this test to fail with the unrelated-message reason. The real policy is restored. - Removed the added source comments to follow the contribution rules. - `node scripts/check-module-boundaries.mjs` and `git diff --check` pass. - `pnpm verify` returns exit 2 because the operator documentation is a global CI input. Required CI must supply the wider coverage. - Local preflight installed dependencies and built the plugin SDK. It could not build Runner Rust binaries because this environment has no `cargo` executable. The focused server tests run without those binaries. - Live Slack and browser tests were not run. This change uses the existing provider message and thread identities. - Required CI passed on `31b1adede2b92bad24b8fa21b8f6fffa9bd26d04`: [PR workflow](https://github.com/paperclipai/paperclip/actions/runs/43035551294), including typecheck, build, server/chat tests, Runner checks, and browser E2E. Current-head Greptile and Paperclip reviews are both 5/5 with no open findings. ## Risks - Existing connections with mention gating on will now accept unmentioned thread replies. This is the intended behavior change. - A missing or invalid thread or message ID does not grant the reply exception. - No schema migration is required. Destination, sender, and company access checks stay in place. - The existing chat integration test file is large. This change updates its current Slack cases to reuse the full delivery and authorization fixture. ## Model Used OpenAI Codex, based on GPT-6, with reasoning and code execution tools. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>canary/v2026.1111.0-canary.10 |
||
|
DottaandClaude Opus 5.5
|
fa3cd6f330 |
fix(native): end provider startup rejections with Retry instead of an ownership hold (#20808)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The Paperclip Runner starts Codex for native runs. On a remote
sandbox such as Daytona, the sandbox is the OS boundary.
> - A Codex planning run on Daytona failed in `session.open`. Codex's
read-only sandbox helper could not set up loopback in the container
(`bwrap: loopback: Failed RTM_NEWADDR`), and Codex loads `AGENTS.md`
through that helper.
> - The server treated that definite provider rejection as an
interrupted transport. Its same-run bootstrap retry went through remote
restart recovery and stopped on `ambiguous_remote_recovery_evidence`.
The task showed "Recovering saved work…" and no Retry, although no
provider work was done.
> - This pull request gives such startup rejections a named, permanent
outcome with a working Retry, and pins ask and standard runs to the
external sandbox policy.
> - The benefit is that a provider startup refusal ends as a clear
failure with Retry, never as an ownership hold.
## Linked Issues or Issue Description
**What happened?**
A Codex planning run on a Daytona sandbox failed in `session.open` with
`error creating thread: failed to load AGENTS.md instructions ... bwrap:
loopback: Failed RTM_NEWADDR: Operation not permitted`. The server
recorded `native_session_interrupted` with recovery mode
`bootstrap_retry`. Thirty seconds later, remote restart recovery blocked
the run with `native_restart_recovery_blocked` /
`ambiguous_remote_recovery_evidence`. No server restart happened. All
default-mode Codex runs on the same image succeeded that day; only the
planning run failed.
**Expected behavior**
A provider startup failure with no provider work must end as a clear
failure with a working Retry, or get a bounded automatic retry that also
ends with Retry. It must never become an ambiguous-ownership hold.
**Steps to reproduce**
1. `cd server && npx vitest run
src/services/native-runtime/native-startup-rejected-retry.integration.test.ts`.
On master the deterministic case is `retryable_failure` /
`native_session_interrupted`.
2. `npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t "provider
itself rejected"`. On master the new code is not produced.
**Paperclip version or commit**
master at
|
||
|
DottaandClaude Opus 5.5
|
6dd8611a2f |
fix(queue): keep tool action outcomes out of the steering queue (#20582)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents ask a person to approve a governed tool action (for example, create a Notebook page) before the tool runs > - Approve means run: the tool runs as soon as the person approves it, and the approval card then shows Executed, Failed or Declined > - After the tool runs, the server wakes the agent with the recorded result. When another turn is active, that wake waits behind it > - The composer queue lists waiting interaction receipts as messages. The result wake carries the approval's interaction id and status, so the queue shows "Accepted: Approve tool action" with an Interrupt control, after the tool already ran > - People read this row as an approval that is still not done, and Interrupt stops their current turn only to deliver a tool result > - This pull request keeps tool action outcome wakes out of the composer queue and out of its steer, interrupt and discard actions > - The benefit is a steering pane that shows only messages the person can act on, while the agent still receives the tool result after the current turn ## Linked Issues or Issue Description **What happened?** After a person approves a governed tool action, the tool runs and its card shows Succeeded. When the person then sends a new message, the composer queue shows a row "Accepted: Approve tool action" with Interrupt and Discard controls. The row stays until the active turn ends. **What did you expect?** The steering pane does not show the tool result as a queued message. The approval card already shows the outcome. The agent receives the result after the current turn. **Steps to reproduce** 1. Let an agent request approval for a governed tool action (`request_confirmation` with `payload.toolAction`). 2. Approve it. The tool runs and the card shows Succeeded. 3. Send a new message so a new turn starts before the result wake runs. 4. See "Accepted: Approve tool action" in the composer queue. ## What Changed - Add `isToolActionOutcomeReceipt` in `server/src/services/queued-interaction-response.ts`. It identifies an interaction receipt wake that carries a tool action (`toolAction` identity or `toolActionRequestIds`). - `findQueuedCommentWake` in `server/src/routes/issues.ts` skips these wakes, so `GET /issues/:id/queued-comments` does not list them. The payload projection adds only the tool action's `actionRequestId`, `toolName` and `toolActionRequestIds`, never its result text. - The locked by-id queue lookup (steer, interrupt, discard) rejects these wakes with `queued_comment_not_pending`, so a stale client cannot interrupt a turn for a tool result. - Delivery, stranded-queue recovery and the handback hold (`readQueuedInteractionResponse`, `hasQueuedInteractionResponse`) do not change. The agent still receives the result. - Add route tests for accepted and declined tool action outcomes. They check that the queue is empty, the wake stays `deferred_issue_execution`, and the issue still has owed interaction input. ## Verification - `cd server && npx vitest run src/__tests__/issue-queued-comments-routes.test.ts src/services/issue-queued-comment-queue.test.ts`: 137 of 137 tests pass. The new tests fail without the fix (the queue lists the receipt). - On a heavily loaded machine (load average about 25), the embedded Postgres `afterAll` teardown timed out once at 30 s after all tests passed. This is not related to the change. - `cd server && npx tsc --noEmit -p .`: clean. ## Risks - Low risk. The change only filters the user-facing queue view and its by-id actions. - A tool action outcome wake no longer appears as an Interrupt target. Before, Interrupt could deliver a tool result into a new turn early. Now the result arrives when the current turn ends, as for other system deliveries. - #13160 refactors the same steering path. One of the two PRs may need a small rebase. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, browser inspection of staging). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
123ee31552 |
fix(ui): omit task IDs from parent breadcrumbs (#20804)
## Thinking Path > - Paperclip helps people manage AI agents and their work. > - Task headers show a path from the parent task to the current task. > - The parent link adds a task number before its title on desktop. > - That extra number makes the parent title change when users open the task. > - This pull request shows the parent title without its number at every width. > - Users can follow the parent link with a consistent task title. ## Linked Issues or Issue Description Related: #20366 added parent task navigation. #10933 added the current task number. **What existing behavior does this improve?** Parent links in task header breadcrumbs. **Current behavior** A desktop parent link includes its task number. A private parent chip also shows its number on desktop. **Proposed behavior** Show only the parent title at all widths. Show Private for a private parent. Keep the current task number, title limits, and responsive truncation. **Reason and benefit** The parent title stays the same when the user follows the link. ## What Changed - Remove the parent identifier from the task page breadcrumb data. - Remove the unused viewport dependency from the breadcrumb effect after review. - Show the Private label at all widths for a private parent. - Update component tests, page navigation tests, and Storybook assertions. - Document title-only parent links in DESIGN.md. ## Verification - Six updated page regressions fail before the fix and pass after it. - All 285 focused BreadcrumbBar and IssueDetail tests pass. - UI typecheck and token gates pass. - Catalog validation and all 30 map/catalog tests pass. - Storybook build passes. All ten Chromium stories finish render/play with no page errors. - Browser checks cover 1440, 768, 390 and 320 pixel widths, light and dark themes, private parents, and keyboard parent navigation. - Parent numbers are absent at every width. Both long titles truncate on phones. The current task number stays visible, and the page has no horizontal overflow. - Private labels stay the same during desktop to phone to desktop resizing. Seven screenshots are saved as task evidence. - These checks mount the production header with fixture data. Page tests use mocked APIs. No live backend authorization or pixel baseline comparison is claimed. - [CI run 43020142521](https://github.com/paperclipai/paperclip/actions/runs/43020142521) passed on |
||
|
DottaandClaude Opus 5.5
|
4c04c20c9e |
fix(recovery): wake a parent whose refused completion waited on dependencies that later clear (#20803)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A parent task that waits on blocker tasks resumes through the `issue_blockers_resolved` wake > - A periodic dependency sweep sends that wake when a blocked task's dependencies become ready and no wake covers the current blocked cycle > - The sweep skips a task that its own agent blocked, because that agent waits on something outside the blocker graph > - A native completion that Paperclip refuses for unresolved dependencies is also recorded as a block by the agent's own run (`durable_dependency_blocker_bound`). The sweep treated it as a self-block. It woke the task only if a blocker later completed or finalized on its workspace > - When readiness cleared another way, nothing woke the task. It stayed `blocked` with no next action > - This pull request makes the sweep treat that refusal as a dependency wait, and count a "Recover saved files" finalize without a workspace as the blocker settling > - The benefit is that a blocked task is not a silent dead end: it resumes once, automatically, when its dependencies clear ## Linked Issues or Issue Description Refs #20728 **What happened?** A parent task had three done child tasks that block it. One child was still listed as "saving its files", because a "Recover saved files" finalize had no workspace. The parent's agent tried to finish. Paperclip refused the completion because dependencies were unresolved and set the parent `blocked` (`durable_dependency_blocker_bound`). After #20728 was deployed, the parent's dependencies were ready. But the task stayed `blocked` with no wake, no Retry, and no next action. A person had to post a comment to wake it. **Expected behavior** When the last blocker of a blocked task clears, by status or by finalize settlement, Paperclip wakes the assignee once. This also applies to tasks already blocked before a deploy that changes readiness. **Steps to reproduce** 1. Block a task on a done blocker whose workspace finalize is still pending. 2. Let the assignee agent's native run report the task done. Paperclip sets the task `blocked` with `durable_dependency_blocker_bound`. 3. Make the blocker settle without a later blocker completion or a finalize on its workspace (for example, a recovery finalize with no workspace, or a deploy that changes the readiness rule). 4. Run the dependency sweep. Before this fix it reports `selfBlockedSkipped: 1` and sends no wake. ## What Changed - `server/src/services/recovery/service.ts` (`wasBlockedByAssigneeAgent`): a native block with reason `durable_dependency_blocker_bound` is not a self-block. The dependency owns its wake. Other native blocks and agent blocks keep the self-block rule. - The same function counts a later succeeded `workspace_finalize` with `owningService: workspace_recovery` and no workspace as the blocker settling, which matches the readiness rule from #20728. - `server/src/__tests__/heartbeat-issue-liveness-escalation.test.ts`: regression tests for the refused-completion shape (one wake, no second wake), the recovery finalize without a workspace, and a safety case for a different native block reason. - `server/src/services/workspace-recovery-dependency-finalize.test.ts`: a characterization test that a real "Recover saved files" signals the dependency sweep for the company. This path already existed; the test keeps it. - `doc/execution-semantics.md`: documents when the dependency sweep wakes after the last blocker clears. ## Verification - Red (unfixed `service.ts`): `npx vitest run src/__tests__/heartbeat-issue-liveness-escalation.test.ts -t "refused for unresolved|still holds a native block|recovery finalize without a workspace"` fails 2 of 3 tests with `selfBlockedSkipped` 1 instead of 0. - Green (fixed): `src/__tests__/heartbeat-issue-liveness-escalation.test.ts` passes 44 of 44. - Related suites pass: `src/__tests__/issue-dependency-wakeups-routes.test.ts`, `src/__tests__/heartbeat-dependency-scheduling.test.ts`, `src/services/workspace-recovery-dependency-finalize.test.ts`, `src/__tests__/issue-recovery-dependencies.test.ts` (43 tests). Server `tsc --noEmit` passes. - `pnpm verify` exits 2 because a doc change is a global CI input. Required CI supplies broad coverage. - Staging evidence (read only): the affected parent's activity shows `native-status-committer` set it `blocked` with `durable_dependency_blocker_bound`, and no later blocker completion or finalize. ## Risks - Previous-release story: no persisted shape changes. Tasks already `blocked` with `durable_dependency_blocker_bound` and ready dependencies are picked up by the existing dependency sweep after deploy (it visits every blocked task at startup and on recovery signals). No migration. - Bounded: the wake uses the cycle-aware ready-state key, so one blocked cycle gets one wake. The sweep keeps its live-run, queued-wake, pending-interaction, and pause-hold checks. - No loop: if the woken agent blocks the task again while its dependencies are ready, the reason is not `durable_dependency_blocker_bound`, so the self-block rule applies. - The recovery finalize rule only applies to a finalize on the blocker task itself with `owningService: workspace_recovery` and no workspace. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, code search, test execution, read-only staging API inspection). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
556b249897 |
fix: preserve exact Markdown in native document writes (#20605)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agents save Markdown documents through native tools and REST routes. > - The native write_document handler trims the supplied body. > - This removes final newlines and can change Markdown whitespace. > - This pull request preserves the original body after required-content validation. > - Agents can now write and read the same Markdown string across these paths. ## Linked Issues or Issue Description **What happened?** The native write_document tool removes leading and trailing whitespace. A body with one final newline loses that newline on read-back. The REST document PUT and write_task_document preserve it. **Expected behavior** The native write_document tool must preserve the supplied Markdown body exactly. Required-content checks and key and title normalization must stay in place. REST-backed paths retain surrounding whitespace and keep their existing escaped-line-break conversion for single-line payloads. **Steps to reproduce** 1. Write a disposable document through write_document with a final newline. 2. Read the document and compare the full body with the supplied string. 3. Repeat through write_task_document and REST. These paths preserve the string. Related work: Refs #18824. That change aligns document keys. This change preserves body strings. A search of open and closed PRs found no duplicate fix. ## What Changed - Use a separate untrimmed required-string helper for the document body. Keep the existing requiredString callback signature and all other field normalization. - Add real tool-handler, database, and authenticated HTTP tests for create, update, read-back, and replay. Cover leading spaces, hard breaks, blank lines, final newlines, no final newline, Unicode, and mixed line endings. - Keep blank and invalid bodies rejected. Document whitespace preservation and the existing REST escaped-line-break exception. - Fix the Runner support archive-scanner shutdown defect exposed by CI: rejected ZIP scanners now receive SIGKILL so an ignored SIGTERM cannot leave publication waiting for process closure. Keep all secret and archive limits unchanged. A real child-process regression fails before the change and passes afterward. - Address all three initial review findings: callback type compatibility, contribution comment policy, and the REST documentation limit. ## Verification Final tested head: `c9ff8cf3f83aa19e5d8313c65816a3a2e6562368`. - All applicable [CI checks](https://github.com/paperclipai/paperclip/actions/runs/43019885807) passed. This includes typecheck, build, Runner support, Rust, native Runner, general tests, serialized suites, all eight E2E shards, and the required aggregate gates. - [Greptile review](https://github.com/paperclipai/paperclip/runs/130520170309) passed on this head with confidence 5/5. All three review threads are resolved. The independent Paperclip review also reports 5/5 with no findings. - Red at `7368a0b15c2f02107778a961144d86e2eb59a952`: `pnpm --dir server exec vitest run src/services/native-runtime/task-document-body.integration.test.ts` failed because write_document removed the final newline. Cross-task and REST cases passed. - Green on the final source: 176 tests passed across the document body, document key, native tool authority, task scope, document service, and document route suites. The tests cover create, update, exact read-back, replay, blank rejection, revision guards, and company scope. - The ZIP shutdown regression failed before repair with “scanner did not stop”. After repair, 54 tests passed across zip-inspection-shutdown, private-reporter-artifacts, manual-private-cell-report-assets, and private-evidence-budget. The regression uses a real process that ignores SIGTERM. - Earlier CI runs had two archive-related timeouts. The source repair above replaces blind reruns. The Runner support suite and aggregate verify gate now pass. - A disposable live native document reproduced the original whitespace loss. REST preserved the same string. The fixed handler passed local integration tests. The running service has not been deployed. - Local environment: Linux x64, Node 25.9.0, Vitest 5.0.3, and embedded PostgreSQL. Local Rust preparation was unavailable because cargo is absent. Final-head CI passed Rust verification. `pnpm verify --explain` requested broad CI because evidence.ts is a global input. Required CI supplied that coverage. ## Risks Low risk. The change retains required-content validation. The product change affects document body whitespace handling. The CI repair forcefully stops only already-rejected archive scanners; it retains fail-closed publication and all existing scan limits. It does not change schema, authorization, revision guards, receipts, or persistence. ## Model Used CI repair continuation: GPT-6.1, using repository tools and code execution. OpenAI Codex, GPT-6 (the exact serving model ID and context window were not exposed in this session). Used reasoning, repository tools, and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>canary/v2026.1111.0-canary.9 |
||
|
DottaandPaperclip
|
84fc67317e |
feat(runner): export authorized read responses as artifacts (#20606)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native agents use authorized API reads to get task evidence. > - Small responses return inline JSON, so an agent must copy each document into a local helper. > - Copying changes bytes and uses model output for data transport. > - This pull request adds an explicit artifact export for non-sensitive GET reads. > - Helpers can download the exact response bytes and check their identity. ## Linked Issues or Issue Description **What existing behavior does this improve?** The native `call_api` response capture path. **Current behavior** Small JSON responses have no artifact reference. Local helpers need the model to copy the JSON data. **Proposed behavior** `saveResponse: true` saves a complete authorized GET response below or above the inline threshold. Zero-byte success returns an explicit empty receipt without storage. The receipt returns an artifact ID, byte size and SHA-256 for `download_artifact`. Other callers keep the current behavior. Related work: #17393 covers saved app-tool result size. #18105 covers mutation response paging. This change adds an explicit read export. ## What Changed - Add GET-only `saveResponse` validation and capture through the current artifact store. - Refuse sensitive operations, denied reads, partial responses and combined text paging. - Keep route authorization, actor visibility, storage budgets and download integrity checks. - Update the native tool contract, generated catalogs and API guide. - Add boundary and real-route byte comparison tests. - Preserve saved run state. This is an optional input field; old calls and receipts keep their current shape. No schema migration or identity reset is needed. The seeded catalog fixture is generated. ## Verification - Final published head `d47b52e1bf412418e24768f865cc139097a6d068`: [typecheck, build, all tests, Linux Rust and aggregate verify passed](https://github.com/paperclipai/paperclip/actions/runs/42918883477); [Windows Rust passed](https://github.com/paperclipai/paperclip/actions/runs/42918883348); [quality gate passed](https://github.com/paperclipai/paperclip/actions/runs/42918883016). Greptile completed successfully at this exact head, 5/5; all five review threads resolved. Automated Paperclip review remains queued due to a failed reviewer run; no final verdict is claimed for that service. - Current head: `d47b52e1bf412418e24768f865cc139097a6d068`. Merged current master to refresh GitHub's stale base snapshot; the PR now contains exactly nine intended artifact transport files, with no lockfile or UI changes. Fresh final-head CI/review is required. - Previous head `986ad87490fff93cfec313ba6fd56d3dc0bb665b` passed typecheck, build, Rust verification, artifact/native suites and completed test shards; remaining shards were still running when the base snapshot was refreshed. Greptile was 5/5, with one unrelated master UI finding resolved as outside the corrected PR diff. - Review fixes verified at `986ad87490fff93cfec313ba6fd56d3dc0bb665b`: 2,135 checks passed across the five focused suites; Runner TypeScript build and regenerated contract/catalog checks passed. This head replays the original implementation on current master and retains the original PR commit in its history. - Review red/green at replayed baseline `da3a1d4de9`: empty HTTP 200/204 tests failed with real storage's `File is empty`, then passed with zero new assets or quota reservations. Error-body tests failed without size/truncation fields, then passed. - Empty successes now return `empty: true`, `byteSize: 0`, `data: null` without storage. Nonempty JSON results such as `[]` export normally. Error receipts report byte size and truncation. Docs and tool guidance state the existing 10 MiB download limit and paged text alternative. - Previous-head CI typecheck/build/tests passed except the Rust process-lineage cleanup test (`unattributed_orphan`); final-head CI and fresh review remain required. - Red at `7368a0b15c`: the small-document export test failed with `Invalid call_api arguments` before production edits. The same check passed after the change. - Final post-rebase run at `6e7a35ed6a59bb74f4218312d9f10557099433c3`: all 2,124 checks passed across the API, capture, artifact boundary, real-route export and artifact download suites. - Real-route export test passed: HTTP bytes, stored bytes and downloaded bytes were equal. Foreign-company access and forged download metadata were rejected. - Existing API integration and artifact suites passed. Nearby API integration checks also passed (59 passed, 2 skipped). - Disposable native helper scan passed: 12 existing documents, 15 GET requests, zero mutations, 73,919 bytes, 17.985 seconds. All files passed size/hash checks. The bridge passed file references only. Production token savings were not measured. - Runner TypeScript build and generated contract checks passed. - `pnpm verify` returned exit 2 and required broader CI coverage. Local preflight stopped at missing Cargo after dependency and SDK setup. The local server typecheck stopped during the cycle layer without a diagnostic; the container recorded an OOM kill. This check has no pass verdict. Required CI must cover typecheck, build and test at the final head. ## Risks - Each explicit export uses the current capture budget and storage admission. Frequent small exports can reach that budget. - An artifact holds one HTTP response. Callers must still follow list pagination. - No database or saved-state migration is required. The option does not change default response behavior. ## Model Used OpenAI `gpt-6.1-sol`, through Codex in Paperclip Runner. Used reasoning, shell execution and native tools. The active agent configuration supplied this exact model ID. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
bc453342ef |
fix(native): start a cold server from active database checkpoints (#20792)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runs save their controller state in complete encrypted database checkpoints. > - A rolling update starts a replacement with an empty local home while the old server still owns active remote work. > - Startup required a local controller marker before it checked the active database authority. > - The replacement could report `native_controllers_require_migration` even when the checkpoint, workspace and live owner were valid. > - This change uses the existing complete checkpoint proof when the local marker is absent. A marker that exists keeps its strict validation. > - The benefit is that a cold replacement can start while supported remote work continues. ## Linked Issues or Issue Description **What happened?** A volume-free replacement failed startup with `native_controllers_require_migration`. The old server had a local external-control marker and a valid active database checkpoint. The replacement had the checkpoint but no local marker. **Expected behavior** An absent local marker must not make a complete, valid database checkpoint into a local migration obligation. Missing, corrupt or mismatched durable evidence must still refuse startup. **Steps to reproduce** 1. Save a full native checkpoint with its real checkpoint writer. Keep its source run active with a live generation-one owner and its bound remote workspace. 2. Remove only the local controller marker to represent a cold replacement home. 3. Inspect native controller obligations. Before this change, the inspection returns true. **Paperclip version or commit** Reproduced on `eb75c9bb855ce83915256efc9d6b308a272cdeff`. The fix is based on `c44a422607f9f08c39ad98f9c830fe6e3dba8dc2`. **Deployment mode** Volume-free server with native remote execution and database checkpoint persistence. ## What Changed - Let an absent local marker use the existing full checkpoint, run, session, workspace and owner checks. - Keep validation of every present marker and the final file reread that detects a change during inspection. - Add a cold-home variant of the existing real PostgreSQL active-checkpoint regression. Its live source owner has a different boot ID from the inspecting process. - Run the same refusal controls for missing checkpoint parts, invalid identity, stale leases, changed task/workspace bindings and invalid local markers. ## Verification - Genuine RED: the cold-home case failed on unchanged production code with `expected true to be false` after the real checkpoint was saved and only its local marker was removed. - Rebased GREEN, Node 24: `node node_modules/vitest/vitest.mjs run --project @paperclipai/server server/src/services/native-runtime/native-controller-migration.integration.test.ts -t 'offline native controller migration \(active_(durable|durable_cold|warm)\)'` — 3 passed, 53 unrelated cases filtered out. These controls use disposable real PostgreSQL and the canonical encrypted checkpoint writer/reader. - Server layered TypeScript check: `cd server && node scripts/typecheck-layers.mjs` — PASS for all source and test layers. - `node scripts/check-module-boundaries.mjs` and `git diff --check` — PASS. - [Required CI](https://github.com/paperclipai/paperclip/actions/runs/43003635878) passed on `7c5cb8fc16f3f7e3bcfec0ce10bf0b43d638d09b`. All 60 check runs are terminal: 55 success, 4 skipped and one neutral 5/5 review. Greptile is 5/5 with no review threads. Live rolling-update acceptance remains a separate delivery check. ## Risks - No schema or saved-state format changes. State saved by the previous release uses the same checkpoint reader. - The absent-marker path still needs a complete valid checkpoint and the existing identity, workspace and owner proof. Present legacy or malformed markers still refuse. - This changes eligibility only. It does not take ownership, launch work, copy state, release a lease or alter admission. - No command or configuration changes need operator documentation. ## Model Used OpenAI Codex, GPT-6, with reasoning, tool use and code execution. The host does not expose a more specific model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal issue id - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>canary/v2026.1111.0-canary.8 |
||
|
DottaandClaude
|
c44a422607 |
fix(native): retry a follow-up whose own server stopped serving before its remote runner (#20786)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The native runtime runs each agent task through a Runner. On a graceful shutdown (SIGTERM), the server closes its runtime API listener first, then drains runs. > - A run that is still preparing can reach the runtime API lookup after the listener closed. It then fails with `local_runtime_api_listener_unavailable` before it reaches its Runner. > - For a remote (sandbox) Runner, the executor already settles this failure as a same-run bootstrap retry, so the next server restarts it with no manual Retry. > - A follow-up run is different. Admission rebinds the settled predecessor's provider session into the follow-up's run as its checkpoint. The recovery evidence saw "a checkpoint exists" and called the failure `ambiguous_state`. That blocked the task and the user had to click Retry. > - This pull request recognizes that carried checkpoint. When this server's own listener closed, the run has no Runner checkpoint and no provider event of its own, and a settled predecessor saved that exact provider session, the failure is a same-run bootstrap retry. > - The benefit is that a graceful restart during a follow-up's preparation no longer needs a manual Retry. ## Linked Issues or Issue Description **What happened?** In a Daytona restart-chaos wave (codex, remote sandbox Runner), a follow-up run was created. About 2 s later a graceful SIGTERM drain began. The draining server kept preparing the run. `resolveLocalRuntimeApiUrl()` in `createRunnerdBackendWithinSessionClaim` threw `LocalRuntimeApiListenerClosedError`. The run failed with `local_runtime_api_listener_unavailable`. The task went to `blocked` with a board-owned recovery action, and it needed a manual Retry. The manual Retry worked. **Expected behavior** No provider work started for the run. The next server must pick it up automatically. **Steps to reproduce** 1. Finish a remote native run for a task, so that its provider session is saved. 2. Comment on the task to start a follow-up run. The follow-up carries the saved provider session as its admission checkpoint. 3. Send SIGTERM while the follow-up is in preparation, before it reaches its Runner. 4. The follow-up fails with `ambiguous_state` and the task is blocked. **Paperclip version or commit** `master` at `62b7733161` (wave) and `eb75c9bb85` (this branch base). Refs #20514, #20721 ## What Changed - `native-session-executor.ts` (`nativeProviderRecoveryEvidence`): add one narrow case before the generic checks. When all of these are true, return `bootstrap_retry` (with `checkpointExists: true` and `providerSessionEstablished: false`): - the source failure is `local_runtime_api_listener_unavailable` (this server stopped serving); - the target is remote, the run has the current preparation fencing version, and the run has no Runner controller checkpoint; - the run has no provider event; - the run checkpoint has no cursor, turn, terminal, result, or pending request of its own; - a settled earlier run of the same native session saved that exact provider session. - `native-restart-recovery.ts`: export the existing `carriesSettledPredecessorSession` proof, so that the executor uses the same proof that restart recovery uses for rebound sessions. - `native-controller-recovery.integration.test.ts`: add a regression test. It also shows that the verdict stays `ambiguous_state` for a different failure code, for an unknown provider session, for a checkpoint with its own turn, and for a run with a provider event. ## Verification - Red on master: `cd server && npx vitest run src/services/native-runtime/native-controller-recovery.integration.test.ts -t "own server stopped serving"` returned `recoveryMode: "ambiguous_state", providerSessionEstablished: true`. - Green with the fix: the same file passes (25 tests). - `npx vitest run src/services/native-runtime/native-startup-retry.integration.test.ts src/services/native-runtime/native-runner-restart-preparation.integration.test.ts`: 43 passed (the existing closed-listener bootstrap-retry and ambiguous cases are unchanged). - `cd server && pnpm typecheck`: passed. - Not run: a live restart-chaos wave. ## Risks - Low risk. The new case applies only to the closed-listener failure on a remote target. It needs durable proof that the run has no Runner checkpoint and no provider event, and that the provider session belongs to a settled predecessor. All other cases keep their earlier verdict. - If the retry later finds different evidence, the existing restart recovery and resumption checks re-prove it and fail closed. The task then keeps a working Retry. - Upgrade safety: no persisted shape changes. The proof reads only rows that older servers already write (`sessionCheckpoint`, `nativePreparationFencingVersion`, Runner checkpoints, run events). A run that an older server prepared without the fencing version keeps its earlier verdict. - No waits are added before `turn.start`. The new queries run only on the failure path. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude
|
59ded526be |
fix(native): ask once to re-check Paperclip before resting blocked on its own stall (#20779)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runs report their result through a completion report. The server checks the report before it accepts it. > - When the Paperclip server pauses or restarts, the agent's Paperclip tool calls stay unanswered. They return later, all together. > - The agent can read that silence as an outage. It then reports a blocker owned by a "Paperclip ... service" with reasonCode `external_service_unavailable`. > - The server accepts that report. The task rests blocked with no Retry. Paperclip's own stall stops the task. Our rule says that must not happen. > - This pull request rejects that blocker report one time per run. The correction tells the agent that Paperclip is reachable now and to re-check. > - The benefit is that the agent continues in the same turn and reports the real outcome. ## Linked Issues or Issue Description **What happened?** In a restart-chaos test, the server was paused (SIGSTOP) for about 2 to 5 minutes while a Codex agent saved a task document. `write_document` and `read_document` did not return during the pause. After the pause, the save had succeeded, but the agent reported a blocker: owner "Paperclip document service", reasonCode `external_service_unavailable`. The run succeeded with disposition `blocked`. The task rested blocked with `canRetry=false` and no automatic recovery. A second test (two long server holds, then a same-run resume) gave the same result. **Expected behavior** A Paperclip stall or restart must not leave a task stuck. The agent must re-check Paperclip after it is reachable and finish the task. **Steps to reproduce** 1. Start a native Codex task that saves a task document. 2. Pause the server process with SIGSTOP for about 2 minutes during the save, then resume it. 3. The agent reports a blocker owned by a Paperclip service. The task rests blocked with no Retry. **Paperclip version or commit** master atcanary/v2026.1111.0-canary.7 |
||
|
DottaandClaude
|
eb75c9bb85 |
fix(native): resend a throttled harness backup upload instead of losing it (#20772)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - After a native run in a per-turn remote sandbox, the server saves a durable harness backup to object storage. The next run restores the provider session from it. > - On staging, two OpenCode runs had an accepted result, but `harness_state.backup.persist` failed. The run log for both says `remote checkpoint failed: Please reduce your request rate.` This is the S3 `SlowDown` (HTTP 503) answer. > - The backup and the OpenCode provider-artifact bundle were uploaded as one stream `PUT`. The S3 SDK cannot send a stream body again, and this path had no retry for a request-rate limit. One `SlowDown` lost the backup. > - The sandbox was then released. The follow-up run had no state to resume and failed with `PRP provider resume state is unavailable` (class P, see #20757). > - This pull request uploads both objects from their encrypted scratch file, as workspace snapshots already do. The existing throttled-retry path then sends them again. > - The benefit is that a short request-rate limit no longer loses the only copy of a provider session. ## Linked Issues or Issue Description Refs #20757 (the failure this causes on the next run). Refs #19917 (added the file-source upload path that this change uses). **What happened?** 1. An OpenCode run in a per-turn Daytona sandbox finished, and its result was accepted. 2. The server uploaded the encrypted harness backup with `storage.putFile` and a stream body. 3. Object storage answered `503 SlowDown` ("Please reduce your request rate."). Nothing sent the request again. 4. `harness_state.backup.persist` failed, and `native.post_result_failure` was recorded. 5. The next run for the same chat task started in a new sandbox and could not resume the provider session. **Expected behavior** A request-rate limit slows the upload down. It does not lose the backup. **Steps to reproduce** 1. Point the storage service at an S3 endpoint that answers the first `PUT` of each object with `503 SlowDown`. 2. Call `persistNativeHarnessBackup`. 3. On `master` it throws `SlowDown: Please reduce your request rate.` **Paperclip version or commit** `master` at `9574065a28`. ## What Changed - `server/src/services/native-runtime/native-harness-backup-storage.ts`: pass the encrypted scratch file as `sourceFile` to `putFile`. - `server/src/services/native-runtime/native-provider-artifacts.ts`: the same for the provider-artifact bundle. - `native-harness-backup-throttle.integration.test.ts` (new, focused file): a real S3 client against a local endpoint that throttles the first `PUT` of each object once. Cases for a Codex backup, and for an OpenCode backup with provider artifacts. Each case also throttles the first read-back of each object once. Each case checks that the backup is current and restores the provider session bytes. ## Verification - `cd server && npx vitest run src/services/native-runtime/native-harness-backup-throttle.integration.test.ts`: both cases fail on `master` with `SlowDown: Please reduce your request rate.` and pass with this change. - `cd server && npx vitest run src/services/native-runtime/native-harness-backup-storage.integration.test.ts`: 68 pass (file unchanged). - `cd server && npx tsc --noEmit -p .`: clean. - `pnpm verify`: it selected the full server suite and failed 16 of 34,997 tests, all local-environment failures unrelated to object storage: missing `.claude/skills/*/SKILL.md` files in this worktree, `ENAMETOOLONG` under the macOS temp directory, and 30 s hook timeouts under load. Required CI is the authority for that suite. ## Risks - Low. A file source also sends a `Content-MD5` header, so the backend rejects any byte it did not receive intact. Workspace snapshots have used this path since #19917. - An upload above 16 MiB now uses the file multipart path. That path has the same verified-part retry. - The read-back check (`getObject`) needs no change: a GET has no body, so the S3 client resends it after `SlowDown`. The test now also throttles the first read of each object once and proves the backup still publishes. - The Rust Runner is unchanged. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code, with tool use (shell, file edit, test execution) and read-only staging API reads. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Nicky Leach
|
4d141cdf64 | Carry plugin task refusals and batch project reads (#20762) | ||
|
DottaandClaude Opus 5.5
|
f125b823f4 |
fix(recovery): release a parent's Retry once a done child's recovered files are saved (#20728)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A parent task waits for its done child tasks to save their workspace files before it runs again (the workspace-finalize barrier) > - "Recover saved files" saves the retained work of a done child and records a successful `workspace_finalize` > - That finalize row had no `execution_workspace_id`. The barrier reads finalize rows by the blocker's workspace, so it never saw the row > - The parent's saved Retry then waited forever with "Saving its files" on a child that had already saved its files > - This pull request lets the barrier accept the recovery's own finalize for the recovered task's finished operations of the same run. It does not make the recovery finalize a workspace-wide save > - The benefit is that recoveries work by themselves: the parent's dependency wait releases and its saved Retry starts on the next dependency pass, with no migration ## Linked Issues or Issue Description **What happened?** A parent task had three done children. One legacy child shared the parent's execution workspace. The operator pressed "Recover saved files" on that child. The recovery saved the files and marked the old run succeeded. The operator then pressed Retry on the parent. The server accepted the Retry, but the parent stayed on "Retry is waiting for a finished task this one depends on to save its files" for more than 15 minutes. `blockerAttention.pendingFinalizeBlockerIssueIds` named the recovered child. The child had no active recovery action and no execution blocker. The workspace was clean (`dirty: false`, `writerRunId: null`). The child's workspace operations showed the cause. The latest operation on the workspace was a succeeded `workspace_recovery`. The recovery's succeeded `workspace_finalize` existed, but with `execution_workspace_id = NULL`. Earlier finalize attempts of that run had failed. `pendingFinalizeBlockerCondition` only looks at operations on the blocker's workspace, so it found no successful finalize after the recovery. **Expected behavior** When "Recover saved files" saves a done child's work, the child must stop holding its dependents. The parent's saved Retry must start without another operator action. **Steps to reproduce** 1. Let a child task finish as `done` with a retained remote workspace whose finalize failed. 2. Run "Recover saved files" on the child, so recovery saves the remote workspace snapshot. 3. Block a parent task on the child. The parent can share the child's workspace. 4. Read the parent's dependency readiness. Before this fix it reports the child in `pendingFinalizeBlockerIssueIds` forever. **Paperclip version or commit** Seen on staging at `a46b9ac7`. Reproduced on `master` at `e039be9202`. ## What Changed - `server/src/services/issues.ts` (`pendingFinalizeBlockerCondition`): the blocker task's latest workspace operation is settled when a succeeded recovery finalize (`owningService: workspace_recovery`) from the same company, task, and run started at or after that operation finished. The operation must be finished (`succeeded`, `failed`, or `skipped`). The recovery finalize can have no workspace (all rows written so far) or this workspace. - The rule is scoped to the blocker task and its run. It does not change the workspace-wide `finalized_at`, so other tasks that share the workspace keep their own barrier. - `server/src/services/workspace-recovery-dependency-finalize.test.ts`: new regression tests that run the real remote "Recover saved files" path, make the child `done`, and block a parent that shares the child's workspace. - `doc/execution-semantics.md`: documents how the recovery finalize settles the dependency barrier. ## Verification - Red (unfixed `issues.ts`): `npx vitest run src/services/workspace-recovery-dependency-finalize.test.ts` fails 3 of 4 tests. The parent stays not ready with the recovered child in `pendingFinalizeBlockerIssueIds`. - Green (fixed): the same file passes 4 of 4 tests: - a real recovery releases the parent in `getDependencyReadiness` and in the issue list `blockerAttention`; - snapshot-fallback shape: the preservation receipt has no workspace and the child's latest operation is the run's failed finalize. The parent is released; - safety: a sibling task that started work on the shared workspace while the recovery waited for teardown still holds the parent; - safety: child work that starts after the recovery still holds the parent. - Focused suites pass: `src/__tests__/issues-service.test.ts`, `src/services/workspace-recovery-admission.test.ts`, `src/services/task-retry.test.ts`, `src/services/saved-bootstrap-continuation.test.ts`, `src/services/remote-workspace-recovery-receipt-fallback.test.ts`, `src/__tests__/issue-dependency-wakeups-routes.test.ts` (414 tests with the new file). - `pnpm verify` exits 2 (a doc change is a global CI input), so required CI supplies broad coverage. Server `tsc --noEmit` passes. ## Risks - Previous-release story: no persisted shape changes. Recovery finalize rows already saved (with no workspace) are read by the new rule. Affected parents become dependency-ready on the next read, with no migration or backfill. The deferred-receipt and dependency backstop passes then start the saved Retry. - The barrier keeps its protection. Unfinished operations, operations from another run, operations that start after the recovery finalize, and other tasks on a shared workspace still wait for their own save. - No schema change and no write-path change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, code search, test execution, read-only staging API inspection). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>canary/v2026.1111.0-canary.6 |
||
|
DottaandClaude
|
1ac8b6a711 |
fix(workspaces): stop re-saving a retained sandbox whose disk is still full (#20758)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - After a native run on a remote sandbox, Paperclip saves the sandbox workspace back to durable storage. When that save fails, saved-work recovery saves the files again from the retained sandbox, with no provider turn. > - A staging census of the last 48 hours (`doc/plans/2026-11-11-staging-failures-vs-chaos-coverage.md`, class F) found saves that failed because the sandbox disk was full. > - #20678 already ends finalization at the first full-disk copy-back and hands the work to saved-work recovery. Recovery then deletes inactive tool caches that it can prove safe, and saves again. > - When that save still found the disk full, recovery recorded a transient `export_interrupted` failure with cause `provider_error`. It then saved again on its automatic schedule (up to 5 retries in 30 minutes). Nothing frees space in a stopped, retained sandbox between those attempts, so each attempt failed the same way. The task showed "Recovering saved work…" and no Retry while this went on. > - This pull request ends the recovery generation at the first such failure with a named cause, `sandbox_disk_full`, and hands it to the board with a Retry. > - The benefit is that the task does not spend 30 minutes on saves that cannot succeed, the user sees why, and one Retry saves the files and starts one run. ## Linked Issues or Issue Description Refs #20678, #20695 (named sandbox disk-full failures for copy-back and restore) **What happened?** A native run on a remote sandbox finished, and its workspace copy-back hit a full sandbox disk. Finalization handed the task to saved-work recovery (#20678). Recovery's own save also hit the full disk. Recovery classified this as `export_interrupted` / `retry_transient` with cause `provider_error`, so it scheduled more automatic saves of the same sandbox. Each one failed the same way. The task showed "Recovering saved work…" with no Retry until the 30-minute window ran out. **Expected behavior** A save that still finds the disk full after the safe cache cleanup is not repeated automatically. The recovery ends at once with the cause `sandbox_disk_full`, and the board gets a Retry. The workspace keeps its previous saved version, and the run's unsaved files stay in the retained sandbox. One Retry saves the files once there is room and starts exactly one run. **Steps to reproduce** 1. Give a native run an accepted result whose copy-back failed (`native_workspace_sync_out_retry_exhausted`). 2. Make every save from its retained sandbox fail with `remote_workspace_sandbox_disk_full`, with no inactive tool cache to delete. 3. Run saved-work recovery. Before this change, it records `export_interrupted` with cause `provider_error` and schedules another save. ## What Changed - `workspace-recovery.ts`: a save that fails with `remote_workspace_sandbox_disk_full` after a confirmed cache cleanup (or with nothing to clean up) now fails the generation with `export_interrupted`, retry class `user_action_required`, and cause `{ label: "sandbox_disk_full", code: "ENOSPC" }`. That class is not retried automatically, and it is not a permanent-source class, so recovery never falls back to the last save and never drops the run's unsaved files. The next sweep hands the hold to the board. - A cleanup whose outcome is unknown (lost acknowledgement, interrupted journal) keeps the existing scheduled retry, because its journal resumes on the next attempt and can still free space. - `retained-provider-cache.ts`: `reclaimInactiveNativeProviderCache` returns `{ changed, confirmed }` instead of a boolean, so the caller can tell "nothing to delete" from "outcome unknown". It has one caller. - `system-recovery-state.ts`: the board handoff text for this cause says that the sandbox disk is full, that Paperclip deletes only inactive tool caches it can prove safe, and that Retry saves the files again. - New `workspace-recovery-disk-full.test.ts` covers the whole path: one failed attempt, no further automatic saves, the board handoff text, the previous saved version unchanged, the agent's files still in the sandbox, no unfinished in-sandbox capture attempt, a Retry offered in the task presentation and blocker, `POST /issues/:id/retry` accepted (202), the files saved after Retry, and exactly one run started (also with a replayed dispatch). - `doc/execution-semantics.md`: documents the behavior. ## Verification - Red on master: `cd server && npx vitest run src/services/workspace-recovery-disk-full.test.ts` fails. The recovery state is `scheduled` with cause `provider_error` instead of `exhausted` with cause `sandbox_disk_full`. - Green on this branch: the same command passes. - Related suites pass: `workspace-recovery-retry-copyback.test.ts` (79 tests, including the 8 cache-cleanup resume tests that need the scheduled retry after an unconfirmed cleanup), `system-recovery-state.test.ts`, `workspace-recovery-capture-policy.test.ts`, `workspace-recovery-fallback.test.ts`, `workspace-recovery-stop-proof.test.ts`, `native-run-finalizer-telemetry.test.ts` (167 tests in all). - `npx tsc --noEmit -p server/tsconfig.json` is clean. - `pnpm verify`: exit 2 (broader verification required), because `doc/execution-semantics.md` is a global CI input. The focused checks above pass. Required CI supplies the broad coverage. ## Risks - Low. The change touches only the saved-work recovery path when its save fails with the exact full-disk message. - Upgrade-safe: the failure code (`export_interrupted`), the retry class (`user_action_required`) and the cause label all pass the earlier release's recovery schema. An earlier release that reads this state shows its generic "Retry to try again" text. - A full-disk failure no longer gets automatic retries in its 30-minute window. If space became free by itself in that window, the user must now press Retry. Nothing in a stopped, retained sandbox frees space by itself, so this is expected to be rare. - No Runner (Rust) change, so no new sandbox image is needed. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file editing, test execution) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude
|
ea25af759e |
fix(native): name a resume that finds no saved Runner state, so Retry starts a fresh session (#20757)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native run can continue the provider session of the previous run on the same task. In a remote sandbox, the Runner keeps the session state inside the sandbox, and the server saves a harness backup after each run. > - On staging, five runs in 48 hours failed with `native_session_recovery_failed: Error: PRP provider resume state is unavailable` (class P in `doc/plans/2026-11-11-staging-failures-vs-chaos-coverage.md`). > - In each case we read, the previous run had an accepted result, but its `harness_state.backup.persist` failed. Its per-turn sandbox was then destroyed. The next run started in a new sandbox with no Runner state for the session. > - The Runner transport threw an untyped error. The runtime could not tell it apart from a transport drop. A governed-wait or same-session checkpoint ended as a generic `native_session_interrupted` with an "ambiguous state" hold that said not to open a replacement session. > - This pull request types that refusal and lets the driver classify it as `provider_state_incompatible` when nothing is in flight. The existing recovery policy then decides. > - The benefit is that the run ends with the permanent named code `native_provider_state_incompatible`. Retry, or the next message, continues in a fresh provider session in the same workspace. No provider turn or tool call can be repeated, because the follow-up never started a turn. ## Linked Issues or Issue Description Refs #20712, #20723 (the same classification for a moved remote workspace root). Refs #20691 (fixes the `provider_identity_incomplete` part of class P). **What happened?** 1. Run A on a chat task finished. Its result was accepted. Its harness backup failed to save (`harness_state.backup.persist` failed, `native.post_result_failure`). 2. Run B started for the next chat message in a new sandbox. It rebound run A's checkpoint with `providerRecoveryPolicy: allow_replacement_after_governed_wait`. 3. The Runner found no control-plane state and no archived authority epoch. It threw `PRP provider resume state is unavailable` as a plain `Error`. 4. The runtime refused a replacement for the governed-wait policy and failed run B with the generic `native_session_recovery_failed`. The task stayed blocked with an "ambiguous state" hold. **Expected behavior** The run ends with a permanent named code and a working Retry, as it does for other deterministic refusals. **Steps to reproduce** 1. Create a Runner transport that resumes a saved provider session from an empty state directory. 2. Request `thread/read`. On `master` it rejects with a plain `Error`. 3. Recover a saved Codex, OpenCode or ACPX session through the driver when the transport fails that way. On `master` the result has no `failureClass`. **Paperclip version or commit** `master` at `9574065a28`. ## What Changed - `packages/paperclip-runner/src/contracts/harness-driver.ts`: add `NativeRunnerResumeStateUnavailableError` (same message as before) and `isRunnerResumeStateUnavailable`. - `packages/paperclip-runner/src/live/runnerd-codex-transport.ts`: throw the typed error at the three branches where the control-plane state is absent. The branch with no PRP identity and the quarantine branch are unchanged. - `packages/paperclip-runner/src/drivers/codex/codex-app-server-driver-impl.ts`: classify the typed error as `provider_state_incompatible` when the saved session has no turn or runtime request in flight. Extract the in-flight test that the moved-root check already used, without changing it. - `doc/execution-semantics.md`: describe the new classification (additive). - Tests: a transport test (local and external Runner state), driver tests for Codex, OpenCode and ACPX Claude, in-flight cases that stay unclassified, and a server case that shows the blocked task offers Retry and that one click starts exactly one fresh-session run. ## Verification - `cd packages/paperclip-runner && npx vitest run src/backends/native-backend-factory.test.ts -t "no saved state"`: 5 pass. With the driver change reverted, the 3 "reports … as provider state incompatible" cases fail (no `failureClass`). - `cd packages/paperclip-runner && npx vitest run src/live/runnerd-codex-transport-cold-restore.test.ts`: 5 pass. On `master` the error is a plain `Error`. - `cd packages/paperclip-runner && npx vitest run src/backends/native-backend-factory.test.ts src/drivers/codex/codex-app-server-driver.recovery.test.ts src/live/runnerd-codex-transport-cold-restore.test.ts src/native-session-runtime.test.ts`: 410 pass. - `cd server && npx vitest run src/services/workspace-recovery-retry-copyback.test.ts -t "does not continue automatically"`: 5 pass, including the new case. It checks `canRetry: true`, an accepted Retry, exactly one queued successor, and `forceFreshSession: true`. - `cd packages/paperclip-runner && npx tsc --noEmit -p .`: clean. - `pnpm verify`: exit 2 (broader verification required) because `doc/execution-semantics.md` is a global CI input. The focused checks above pass. Required CI supplies the broad coverage. ## Risks - Low. The change only adds a class to a failure that already failed. A checkpoint with `allow_replacement_after_resume_failure` already replaced the session for this failure, and still does. - For `same_session_only` and `allow_replacement_after_governed_wait`, the run now ends with `native_provider_state_incompatible` instead of a retryable `native_session_interrupted`. Same-run retries stop at once. They could not succeed, because the state is absent. - A session with a turn or runtime request in flight keeps the old unclassified behavior. This PR does not change it. - The server code path is unchanged. The Rust Runner is unchanged, so no new runner image is needed. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code, with tool use (shell, file edit, test execution) and read-only staging API reads. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude
|
9574065a28 |
fix(workspaces): hand on a proven-unchanged preparation epoch instead of abandoning it (#20736)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native run on a remote sandbox first restores its workspace's saved version, publishes that root as its dirty epoch, and only then starts work > - A restart that kills the run in that window leaves the epoch dirty. The ownership settle (`settlePreparationOnlyRemoteWorkspaceEpoch`) always abandons it: the root is forgotten and the retry restores the saved version into a new directory > - The saved provider session still names the old directory, which is what #20712 and #20723 now have to handle > - In the common case the killed preparation only restored the saved version and changed nothing. A durable proof for exactly that already exists (`proveRemoteWorkspaceUnchanged`, used when the next admission finds an ended writer) > - This pull request makes the settle use that proof first, and keeps abandoning whenever the proof fails > - The benefit is a stable workspace root across a restart during preparation, with no change for anything unproven ## Linked Issues or Issue Description Refs #20712 **What happened?** In restart-chaos waves, a server kill during a follow-up's preparation (right after the workspace restore published) made the ownership settle abandon the epoch. The retry restored into `<workspace>-<epoch>-<lease>` instead of the published root, and the inherited session checkpoint named a root that no longer existed. **Expected behavior** When durable records prove the killed preparation only restored the saved version, the workspace is handed on at that saved version and keeps its root. Anything unproven is abandoned exactly as before. **Steps to reproduce** 1. Run a remote native task to a saved version, then start a follow-up or continuation run. 2. Kill the server after the follow-up's restore publishes (crash point `remote_workspace.prepare.after_publish_before_provision`). 3. Let ownership reconciliation settle the dirty epoch, then let the retry prepare. ## What Changed - `recovery/remote-workspace-unsaved-files-settlement.ts`: before it abandons a preparation-only epoch, the settle calls the existing `releaseUnchangedRemoteWorkspaceWriterById` (lock the workspace, check the writer, then `releaseUnchangedRemoteWorkspaceWriter`). That releases the writer at its saved version (dirty false, writer cleared, location and root kept, baseline marked released, run reference set to `saved`) only when `proveRemoteWorkspaceUnchanged` holds. The proof requires all of these durable records: - the unchanged-baseline marker for this exact epoch, run and saved version. It is written in the same transaction that publishes the restored root, so a kill mid-restore leaves no marker; - no recorded process, process group, process start or provider session; - no execution-start run event, no native result and no native runner checkpoint (the controller writes one before every runner command); - no workspace operation other than the file-neutral restore, finalize and recovery receipts (provisioning and saves record theirs before they touch files); - every workspace repository already in the saved version. - If the proof fails for any reason, the settle abandons the epoch exactly as before. The existing settle preconditions (run terminal, still `preparing`, no finalization, no work-started event, no save in flight) are unchanged and run first. - `packages/shared/src/crash-point-registry.ts` and `remote-workspaces.ts`: add the crash point `remote_workspace.prepare.after_publish_before_provision`, so the chaos harness can kill exactly at this boundary. It counts only a preparation that published an unchanged baseline (a restore of a saved version), and it is a no-op unless a test arms it. `tests/runner-e2e/source-sdk-public-inputs.json` is refreshed because the registry file is a pinned SDK input. - Test `ownership-reconciliation-report.test.ts`: a proven-clean epoch is handed on with its root kept, with no abandonment receipt, and the writer hold is free for the retry. These cases are still abandoned exactly as before: no baseline (an epoch saved by the previous release), a baseline for another epoch, an already-released baseline, a provisioning operation. A run with a recorded process is still neither handed on nor abandoned by this settle. Previous-release story: an epoch saved by a release that did not write the baseline marker has no proof, so it is abandoned as before. No persisted shape changes; the marker already exists on master. ## Verification - Red: with the release branch disabled, the new test fails on the clean case (the epoch is abandoned). Green with the change. - `cd server && npx vitest run src/__tests__/ownership-reconciliation-report.test.ts src/services/remote-workspace-snapshot-reuse.test.ts src/__tests__/crash-point-call-sites.test.ts` passes. `packages/shared` `crash-points.test.ts` passes. `tsc --noEmit -p server` passes. - Live Daytona waves (claude, workload S1, the new crash point at the continuation's preparation (`:2` before the point was gated to saved restores; `:1` now), image built from master `e039be9202`, one task each): | | before (release disabled) | after | | --- | --- | --- | | Killed epoch 2 | abandoned (`workspace.remote_epoch_abandoned`) | handed on (`workspace.snapshot_skipped_unchanged`, epoch 2) | | Retry root | new root `…-4-ecf1c81e…` | same published root `…-1-33e93b90…` | | Retry `task.prepare` | 4598 ms | 4731 ms | | Retry restore / environment acquire | 1141 / 1381 ms | 1106 / 1478 ms | | Task result | done, no Retry | done, no Retry | | Stopped or orphan sandbox at rest | none | none | Preparation time does not change in this setup. The Chaos Lab environment uses ephemeral leases, so every run gets a fresh sandbox and a full restore with or without this change, and the killed run's sandbox is destroyed either way. The gain is the stable root, which keeps the saved session's working directory valid. ## Risks - Low. The release path is the same one admission already uses for an ended writer, under the same proof. The settle now tries it first. - A proof false positive would keep a root whose files changed. The proof refuses on every record that precedes a write, and on any missing or older-release evidence. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edit, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>canary/v2026.1111.0-canary.5 |
||
|
DottaandClaude
|
62b7733161 |
test(restart-chaos): long mid-tool K15 outages, Runner evidence, and TM1 stalls on Daytona (#20711)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The restart-chaos harness (`tests/restart-chaos/`) restarts or stalls the control plane while real agent tasks run. > - The staging census (#20679, `doc/plans/2026-11-11-staging-failures-vs-chaos-coverage.md`) found two open classes that the harness could not create. In class R, a deploy lands mid-turn, the server stays down for 4–7 minutes, and recovery then finds the remote Runner gone (`runner_remote_process_identity_unavailable`). In class L, the server stalls for about 100 s without a restart, and legacy runs that are still `preparing` end up waiting for a manual Retry. > - K15 restarted at once. No Daytona workload kept a tool running across the fault. TM1 was local-only with a fixed range. Nothing recorded what the remote Runner did while the controller was away. > - This pull request adds a K15 outage hold, a mid-tool workload, sandbox-side Runner evidence, and TM1 stalls on Daytona with a configurable hold. > - The benefit: these staging failure shapes can now be reproduced, and each wave records the Runner's own state. ## Linked Issues or Issue Description Refs #20679 **What happened?** On staging, two failure shapes had no matching chaos scenario: - Class R: a deploy landed mid-turn and the server stayed down for 4–7 minutes. Recovery then found the remote Runner gone. Nothing captured why the Runner exited. - Class L: a 100 s server stall without a restart lapsed every controller lease. Legacy `preparing` runs then waited for a manual Retry. **What did you expect?** The harness should create a long mid-turn deploy outage and a stall without a restart on Daytona. It should also save the sandbox-side Runner evidence. **Steps to reproduce** 1. Use the census report and the staging run records listed in it. 2. Compare them with the harness options before this change: K15 has no hold, TM1 is local-only, and there is no Daytona tool workload. ## What Changed - `--k15-hold-seconds <n|min-max>` (0–600). A K15 stops the server with SIGTERM, keeps it down for the seeded hold, then starts the next deploy generation. The range is drawn only for K15 cells, so existing seeds keep their plans. Storms use it too. - New opt-in workload `T1`. The agent runs one shell loop in the foreground for about 100 s, then saves `TICKS-<nonce>=50`. Its fault window, 35–80 s after launch, lands mid-tool. - New `daytona-runner-logs.ts`, a read-only probe of each wave sandbox through the Daytona toolbox. It runs only on sandboxes with the wave's labels, and only on `started` ones. It records every process, memory, disk, `dmesg` OOM lines, and the tails of `runnerd.stderr.log`, `runner-process.identity` and `runner-state.json`. It redacts known secrets. It runs right after a K15 stop, just before the next deploy starts, and at wave end. Output goes to `results/<wave>/daytona-runner/` and to `runner_logs_captured` in the timeline. The probe never stops, starts or deletes a sandbox, and a failed capture is recorded instead of thrown. - TM1 now runs on `--env daytona`, and `--tm1-hold-seconds <n|min-max>` (10–600) replaces its default 95–150 s range with the same single draw. TM2 and TM3 stay local. - Tests: `k15-long-hold.test.ts` and `tm1-stall.test.ts`. The README is updated. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 45 files and 531 tests pass. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: passes. - Daytona waves, all with T1 and K15: | Wave | Revision and image | Hold | Result | | --- | --- | --- | --- | | `w-20261111T015441-s9310` | master |
||
|
DottaandClaude
|
f25fee3532 |
fix(native): hash runner artifacts and run archive tar off the event loop (#20722)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - One server process prepares native runs, renews every controller lease on timers, and serves every request > - On a run path, the server hashed the 31 MB runner binary with `readFileSync` (more than once per run), hashed provider packs and harness backups synchronously, and ran `tar` with `execFileSync` (timeouts up to 60 s, one with no timeout) > - On a slow disk, each of these calls freezes the whole process. A 103 s stall on a hosted instance lapsed every lease. The last event before that stall was immediately before a synchronous read and hash of the runner binary (see #20694 and #20709) > - This pull request moves those reads, hashes and archive commands off the event loop. It computes the runner binary digest once per boot > - The benefit is that a slow disk delays only the run that waits for it. Turns also start faster, because the runner binary is no longer re-read for every run ## Linked Issues or Issue Description Refs #20694, #20709 **What happened?** On a hosted instance, the server process stopped for 103 s without a restart. Every controller lease lapsed, and two tasks needed a manual Retry. The last run event before the gap was immediately before `matchesControllerRunnerArtifact`, which does `createHash("sha256").update(readFileSync(controllerRunnerBinary))`. Four workspace restores were running at the same time, and file I/O was very slow (a 43 s archive hash, 20 s snapshot decrypts). The trigger is not proven. But each synchronous call below blocks the event loop for as long as the disk is slow. **Expected behavior** File hashing and archive commands on a run path must not block the event loop. A slow disk must delay only the run that waits for it. **Steps to reproduce** 1. Make a remote runner binary slow to read (the test uses a FIFO whose writer waits), or make `tar` slow (the test puts a `tar` wrapper that sleeps first on `PATH`). 2. Call `resolveRemotePaperclipRunnerBinary`, `stageRemoteRunnerDirectory`, or `syncRemoteRunnerDirectoryOut` while a 10 ms timer runs. 3. Before this change, the timer stops for the whole slow step (814 ms, 1,764 ms and 3,379 ms). After this change, it keeps firing. ## What Changed - `server/src/services/native-runtime/off-loop-file-work.ts` (new): - `sha256FileStreamed` - `cachedFileSha256`: a streamed digest, reused while the file's device, inode, size, mtime and ctime do not change - `binaryTargetFromHeader`: reads 64 KiB; reads the rest of the file only when the header cannot answer, so the result equals `runnerBinaryTarget(readFileSync(path))` - `sha256DirectoryTreeStreamed` - `execFileBuffered`: `execFile` with stdin input, `maxBuffer`, `timeout` and `env`. It rejects on a non-zero exit, a timeout or a `maxBuffer` overflow, like `execFileSync`. - `native-session-executor.ts`: - `matchesControllerRunnerArtifact` uses `cachedFileSha256`. Each boot reads the binary once, not once or twice for every run. - The remote Claude daemon target is read from the binary header. - The bundled remote runner binary is checked by `verifiedBundledRemoteRunnerBinary`: the same checks and errors, with a cached digest. - Provider-pack upload verification uses the new `verifyRemoteProviderPackManifest`. It runs the same checks, in the same order, with the same errors. One generator drives both the sync and the streamed readers. - Harness backup build and verify on run paths use streamed digests (`buildNativeHarnessBackupManifestStreamed`, `verifyNativeHarnessBackupStreamed`). The digest format does not change. - `stageRemoteRunnerDirectory` (3 calls), `assertSafeRemoteCheckpointArchive` (3 calls) and the checkpoint extract use `execFileBuffered`, with the same arguments, timeouts and buffer limits. `stageRemoteRunnerFile` uses `readFile`. - `packages/paperclip-runner/src/live/runner-binary.ts`: the default daemon lookup (`resolvePackagedRunnerBinary`, used by the local and remote resolvers) classifies the daemon with `runnerBinaryFileTarget`. It reads the 64 KiB header, not the whole file. - `packages/paperclip-runner/src/live/bundled-remote-provider-pack.ts`: the release-manifest checks now live in one helper, `bundledRemoteRunnerRelease`. The sync `bundledRemoteRunnerBinary` and the server's streamed check both use it. - `native-codex-runner.ts`: - `resolveRemotePaperclipRunnerBinary` reads only the header. - The local `runnerDigest` uses `cachedFileSha256`. - New `verifiedBundledRemoteRunnerBinary`. **Ordering and atomicity.** I checked each converted call for code that relied on the sync call being atomic with respect to other runs: - **Checkpoint copy-back.** It extracts into a private `mkdtemp` directory. The replace sequence (`existsSync`, `rename`, `rename`, `rm`) stays synchronous, so no other run can see a half-installed checkpoint. This function already awaited a remote command before the extract, so this change adds no new interleaving with the target. - **Harness backup verify then restore.** The verify was sync, but the restore after it was already async. The window between them existed before. One owner serializes the turns of a session, so no other turn publishes a backup for that session while a turn verifies it. - **Staging and provider-pack hashing.** These read build-owned, immutable trees, or the private staging copies of a run. ## Verification - Red/green: `pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/off-loop-file-work.test.ts` - With `native-session-executor.ts` and `native-codex-runner.ts` from `origin/master`: 3 failed. The longest timer gaps were 814 ms (runner binary target read), 1,764 ms (directory staging tar) and 3,379 ms (checkpoint list, size check and extract). - With this change: 6 passed. The gaps stay under 400 ms. The same file also checks the digest cache, the header fallback, and the `execFileSync`-like errors (exit code, timeout kill, `ERR_CHILD_PROCESS_STDIO_MAXBUFFER`). - `native-session-executor.test.ts`: - The streamed provider-pack verifier returns the same manifest as the sync reader, and rejects a changed asset tree with the same error. - The streamed harness backup manifest equals the sync one. - The streamed verify agrees with the sync verify, for a valid backup and for a corrupt one. - Run together: `native-session-executor`, `off-loop-file-work`, `native-codex-runner`, `setup-readiness`, `antigravity-connection-probe` (1,653 passed). `run-completion.test.ts` passed all 153 tests. Its `afterAll` embedded-Postgres cleanup timed out locally, which is not related to this change; CI runs it. - `pnpm --filter @paperclipai/server exec tsc --noEmit` passes. ## Risks - **Turn start.** This change adds no wait before `turn.start`; it removes work. The first run of a boot streams the runner binary once. Later runs reuse the digest after one `stat`. - **Cache staleness.** The cache keys on device, inode, size, mtime and ctime. An in-place rewrite changes mtime and ctime, and a replaced file changes its inode, so a changed binary is hashed again. - **stderr.** `execFileSync` wrote a child's stderr to the server's stderr. `execFileBuffered` captures it. Tar warnings no longer appear in server logs; errors and exit codes are unchanged. - **Persisted state.** No persisted shape changes. Old saved state and new code (and the reverse) read the same manifests and digests. - **Not converted in this PR:** small metadata syscalls (`existsSync`, `lstatSync`, `readdirSync`, small JSON reads and writes, renames) in the native state machines. Some of these rely on running synchronously. Also not converted: the sync harness backup verify inside the sync restart-authority checks, and the sync stamp verify in `native-harness-backup-storage.ts`. Both are listed for a follow-up. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude Code with tool use, shell, and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes (code comments carry the contract) - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude
|
94c7c83ce8 |
fix(runner): refuse a moved-root same-session checkpoint by name (#20723)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs on a remote sandbox resume a task's provider session from a saved checkpoint > - #20712 made a checkpoint that names an abandoned remote workspace root decline recovery, so a replacement session opens in the new root > - A checkpoint that requires the same session (an unfinished goal, `same_session_only`) cannot be replaced. It then failed with the generic `native_session_recovery_failed`, which recovery treats as a retryable interruption > - A retry cannot move a saved session to another root, so the failure repeats. The upgrade-safe rule requires a permanent, named code for this case > - Also, the working-directory errors said "Codex" on Claude and other ACPX runs, because one app-server driver serves all of them > - This pull request reports the moved root as provider state that is incompatible, and makes the errors name the agent generically > - The benefit is a named, permanent refusal with a working board Retry, and error text that is correct for every agent ## Linked Issues or Issue Description Refs #20712 **What happened?** After #20712, a `same_session_only` checkpoint with a moved remote root ended with `native_session_recovery_failed: persisted session working directory is not the assigned remote workspace`. That is not a named permanent code, so recovery could retry a run that cannot succeed. On ACPX Claude runs, path errors read "Remote Codex working directory ...". **Expected behavior** The run ends with the permanent code `native_provider_state_incompatible`. Automatic retries stop. A board Retry continues the task in a fresh provider session in the same workspace. Errors do not name Codex on other agents. **Steps to reproduce** 1. Save a native checkpoint with `providerRecoveryPolicy: same_session_only` under one remote workspace root. 2. Assign the run a different sibling root (the epoch was abandoned). 3. Resume the run. ## What Changed - `codex-app-server-driver-impl.ts`: the moved-root decline from #20712 now carries `failureClass: provider_state_incompatible`. The runtime already maps that class: - checkpoint allows replacement (`allow_replacement_after_resume_failure`): a replacement session opens in the assigned root, as before; - checkpoint requires the same session: `NativeProviderStateIncompatibleError`, code `native_provider_state_incompatible`. - The server already treats `native_provider_state_incompatible` as a permanent failure (no automatic retry, board action text "Retry the task to continue in a fresh provider session with the same workspace"). `rebindNativeSessionCheckpoint` returns no checkpoint after that code, so a board Retry starts a fresh session in the same workspace. No server change is needed. - A moved-root checkpoint with an active turn or a pending runtime request now reports `provider_side_effect_pending`. The runtime never replaces it under any policy, and the run ends with the permanent code `native_provider_side_effect_pending` for an operator to reconcile. Before, it threw the working-directory refusal, which recovery recorded as the retryable `native_session_interrupted`. - `harness-driver.ts`: the `PROVIDER_STATE_INCOMPATIBLE` doc now names the moved-workspace case. - `codex-boundaries.ts`: working-directory errors say "Agent working directory", "Remote agent working directory" and "Provider thread response" instead of "Codex". Tests updated to the new text. - Tests: - `native-session-runtime.test.ts`: the real app-server backend with `remote_runner` authority and a moved root, for each recovery policy. Same-session policies end with `native_provider_state_incompatible`, the replacement policy opens a replacement in the assigned root, an in-flight turn ends with `native_provider_side_effect_pending` under every policy, and no transport starts. - `native-backend-factory.test.ts`: the decline carries the failure class, and an ACPX Claude refusal reads "Remote agent working directory does not match the assigned workspace". Previous-release story: no persisted shape changes. Saved checkpoints are classified at recovery time. A saved same-session checkpoint under an old root now ends with the named permanent code instead of a generic one. ## Verification - Red: without the failure classes, the same-session and in-flight runtime cases fail (generic `native_session_recovery_failed`). Before the rename, the ACPX Claude message test fails. - Green: `cd packages/paperclip-runner && npx vitest run src/backends src/drivers/codex src/live/runner-prepared-launch-workspace.test.ts src/native-session-runtime.test.ts` passes (784 tests). - `npx tsc --noEmit -p packages/paperclip-runner` passes. - Existing server coverage for the code: `native-session-resume.test.ts` (no rebind after `native_provider_state_incompatible`) and `native-session-executor.test.ts` (source-code mapping). ## Risks - Low. Error message text changes. A search found no code that matches on the old text outside the updated tests. - Same-session checkpoints with a moved root now stop for a board Retry instead of a generic interruption. That is the intended behavior: a retry cannot succeed. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edit, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude
|
08ff0828d6 |
fix(native): offer Retry at once when the Runner refuses a turn start on a closed session (#20729)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native ACPX run (for example Claude) starts each turn with `turn.start`. When a start is cut off, the Runner reads the sidecar's dispatch journal (#20675). > - If the prompt never left, the Runner keeps the session reopenable, and the same-run resume sends the prompt once. If the prompt may have left, the Runner closes the session, because a resend could duplicate it. > - A closed session cannot reopen for the same run. Every same-run resume got `ACPX session is not open`. The server treated that as a generic interruption and spent three attempts (about 95 s) before the board offered Retry. > - This pull request gives that refusal a closed class in the Runner and a permanent, named failure code in the server. > - The benefit is that the board offers a working Retry on the first refusal, while every recoverable case keeps recovering by itself. ## Linked Issues or Issue Description **What happened?** Restart-chaos wave `w-20261111T011704-s9107`, task t05 (local, claude, workload C1, fault TM2). A freeze cut off a turn start, and the Runner saved the session `closed`. Three same-run resumes each failed with `PRP command turn.start failed: … "message":"ACPX session is not open"`. Recovery stopped with `native_session_retry_exhausted` at 01:25:14, and the board then offered Retry. The Retry run succeeded. **Expected behavior** When the Runner proves the session is closed for this run, the run stops at the first refusal with a named failure, and the board offers a Retry that works. A start that the Runner proved never reached the provider still recovers by itself. **Steps to reproduce** 1. `cd server && npx vitest run src/services/native-runtime/native-closed-session-retry.integration.test.ts` 2. On master, "offers Retry on the first refusal, and Retry starts a new run" fails: the coordinator is `retryable_failure` / `native_session_interrupted` (a same-run retry in 30 s). Refs #20675, #20720 ## What Changed - Runner (`durable/mod.rs`, `acpx_provider_backend.rs`): new closed class `provider_session_closed`. The Runner sets it on these refusals: - a turn start that saves the session `closed` (`ACPX turn start failed closed: …`), but not the rejected-checkpoint case, which stays `prepared` and reopens; - `ACPX session is not open` when the saved lifecycle is `closed` for this run; - the saved turn-end-restart refusal for this run (asserted in `acpx_turn_end_restart`). A session that waits for provider-exit proof can reopen after a restart. Its refusal gets no class. - Server (`native-session-executor.ts`): a `turn.start` failure with that class maps to the new permanent code `native_provider_session_closed`. The server reads the class from the authenticated command result, and from the command-failure text if the error was rewrapped. `nativeSessionFailureDisposition` treats the code as permanent, so the first refusal ends the run as `terminal_failure` with a board owner and a plain next action. - Docs: `packages/paperclip-runner/docs/durable-recovery.md` states the class and the older-Runner behavior. - Tests: - `native-closed-session-retry.integration.test.ts` (new, embedded Postgres) drives `executePaperclipNativeSession` with the Runner refusal. On the first refusal, the board shows `canRetry: true`. `requestTaskRetry` is accepted, and `resumeTaskRetry` starts exactly one new run with a fresh provider session. A start proved undispatched stays `retryable_failure`. - Unit tests in `native-session-executor.test.ts`: both closed refusals, a rewrapped error, an undispatched start, an unclassified older-Runner refusal, and a class on another command. - Rust `acpx_cut_turn_start`: a start cut off before dispatch has no class. A start cut off after dispatch, and the later refusal, carry `provider_session_closed`. ## Verification - Red on master, green here: step 1 above. - `npx vitest run src/services/native-runtime/native-session-executor.test.ts`: 1358/1358 pass. - `cargo test -p paperclip-runner-core --test acpx_park_preparation` (26 pass), `--lib acpx` (94 pass), `--test acpx_provider_turns --test acpx_provider_session --test acpx_warm_attachment` (all pass), `cargo fmt --check` clean. - `npx tsc --noEmit -p server/tsconfig.json`: clean. - Not run: a chaos wave. ## Risks - This changes the Rust Runner. Daytona needs a new runner image before remote runs get the class. Until then, an older Runner sends the refusal without the class, and the server keeps today's bounded same-run retries. That path is unchanged and ends with a working Retry. The server does not trust the text alone, because an older Runner sends the same text for a session that can still reopen. - The class is set only where the Runner's saved state proves that no later start of this run can reopen the session. A false class would cost one automatic resume and show Retry earlier. It never replays provider work. - No persisted shape changes. The class is an optional field of a failed command result. Old saved state resumes unchanged. Nothing new runs before `turn.start`. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude (Anthropic), model ID `claude-opus-5-5`, run in Claude Code with tool use (shell, file edit) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
1143abaf92 |
fix(recovery): give due exhausted holds their own fair lane in the recovery worker (#20725)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The workspace recovery worker saves the files of failed runs. It also re-arms exhausted holds under the rules of #20295, #20569, #20641 and #20648 > - After #20680 and #20661 were deployed to staging, the worker still re-armed none of 29 exhausted holds in 36 minutes. One hold's id sorts in the first percent of all candidates, so the restart starvation that #20680 fixed was not the only cause > - The worker skipped every obligation while any unfinished save was selectable. Discovery is instance-wide. A save that another server owns stays unfinished on every run, so obligations never ran. The obligation pass also put due holds behind hundreds of settled holds whose saved input waits, at two per run. A completed pass never revisited a hold that became due by time alone > - This pull request gives due exhausted holds their own lane, and it shares the worker's slots fairly between unfinished saves, due holds and the obligation pass > - The benefit is that every due hold is reached within a few runs of the worker, whatever its other load. The holds save their files and free their tasks without a person ## Linked Issues or Issue Description Refs #20680 Refs #20661 Refs #20648 Refs #20641 Refs #20569 Refs #20295 **What happened?** On staging, 36 minutes after a deploy that included #20680 and #20661, the workspace recovery worker had touched none of the 29 exhausted holds of the company (22 capture-policy holds and 7 stop-proof or fallback holds). They had no `stopProofRecovery` or `capturePolicyRecovery` marker, no re-arm activity, and `updatedAt` from Nov 2 to Nov 10. The worker did run manual "Recover saved files" requests: one stop-proof hold saved its files 57 seconds after the request. So the worker was alive and the stop proof was available. The holds were not reached. **What did you expect to happen?** The worker reaches each exhausted hold whose re-arm or fallback is due, re-arms it, and saves the files, within a bounded time. **Steps to reproduce** 1. Create a scheduled save whose run's finalization lock another database session holds (another server owns the run). 2. Create an exhausted stop-proof hold and an exhausted capture-policy hold, in the evidence shape that the Nov 4 and Nov 10 releases saved. 3. Start the real workspace recovery worker. 4. The worker retries the busy save on every run and never re-arms either hold. ## What Changed - `server/src/services/workspace-recovery.ts`: `reconcileRetainedWorkspaces` draws candidates from lanes. Each lane is a page that its own cursor advances: unfinished saves; for the worker, exhausted holds whose re-arm or fallback is due; and the obligation pass. Before, the obligation pass ran only when no unfinished save was selectable (the gate at `const unfinished = rows.length > 0; if (!rows.length) { ... }`). - Each lane with work takes one free slot before any lane takes a second. The lane that goes first rotates on each run, so with one free slot (a long local export holds the other) each lane still gets it in one run of three. Actions of a run that one lane took join that run's job without another slot. - The due lane wraps at its tail. A hold that becomes due by time alone (a lease expiry or a re-arm backoff) is reached after the obligation pass completed with no committed change. - `retainedWorkspaceCandidates` takes a `dueOnly` filter (due re-arm or fallback of an exhausted hold). The obligation filter uses the same condition. One `now` is used for the whole query. - The startup sweep (no worker sweep state) also shares its slots between unfinished saves and its general pass. - `doc/execution-semantics.md`: document the lanes and the fairness rule. - New `workspace-recovery-passive-fairness.test.ts` with four real-Postgres regressions. ## Verification - Red on `origin/master` ( |
||
|
DottaandClaude
|
16024ffa2a |
fix(native): a same-boot stall never leaves a run waiting on itself (#20721)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs keep an execution lease. Restart recovery takes over a run only when it can prove that the recorded controller stopped. > - A stall (SIGSTOP) longer than the lease, with no restart, makes the executor lose its own lease. The executor then detaches for "the next controller" and exits. > - The next controller is this same process. Recovery asks the PID probe if the recorded controller is alive. The recorded controller is this process, so the answer is always "alive". > - The task then waits forever: `in_progress`, no Retry, an active run, and an active environment lease. > - A second gap: after back-to-back stalls mid-tool, the executor returns and nothing in this process ever looks at the run again. No terminal state, no recovery and no Retry. > - This pull request lets recovery recognise its own process. When this process no longer drives the run and its own lease lapsed, the prior owner is gone. The existing evidence checks then decide, as for any stopped controller. > - It also makes heartbeat schedule that recovery as soon as such an executor exits. > - The benefit is that a same-boot stall never needs a manual Retry. When provider work may have started, the run fails with a working Retry instead of a wait with no end. ## Linked Issues or Issue Description Refs #20694 (the same class of fix for legacy `preparing` runs), #20529, #20691. **What happened?** Chaos wave `w-20261111T034105-s9330`, task t02 (Daytona, OpenCode, fault TM1). The server process was frozen with SIGSTOP for about 84 s while run `89a3922b` prepared. After SIGCONT, `native.process_start_requested` was recorded and `native.session.execute` failed after 118 ms: the lease renewal found the lease lapsed, and the executor detached and exited. From then on, recovery recorded `native.recovery.transition` "waiting for safe ownership evidence" with reason `controller_still_alive` every 30 s for more than 20 minutes. The run stayed `running`, the issue stayed `in_progress`, no Retry was shown, and the environment lease stayed `active`. Chaos wave `w-20261111T040439-s9331` (Daytona, codex/claude/opencode, T1, TM1). Three stalls of about 90 s came back to back, with less than 1 s between them. Each run was in a tool call. After the second thaw, `native.session.execute` failed for all three runs (lease lost, executor detached). After that, no run had any event: no terminal state, no `native.recovery.transition` and no Retry, for 20 minutes. The in-sandbox turns kept going. A single 90 s stall (wave s9340) stays green, because a later recovery signal happens to find the run. With stacked stalls, that signal can land while the executor is still tearing down, so it skips the run and sets no deadline. **Expected behavior** When this process no longer drives a run that it recorded as its own, recovery takes the run back. No turn reached the runner here, so the bootstrap restarts on the same run. When a turn or runner work may have started, the run fails and shows a working Retry. **Steps to reproduce** 1. Start a native remote (Daytona) run. 2. Send SIGSTOP to the server during preparation, and hold it longer than the execution lease. 3. Send SIGCONT. The executor loses its lease, detaches and exits. 4. See that every recovery scan waits with `controller_still_alive`, and the run never settles. The new tests in `native-runner-restart-recovery.integration.test.ts` ("same-boot controller whose own lease lapsed") reproduce this at the recovery layer. ## What Changed - `claimNativeRestartRecoveries`: when the recorded controller is this exact process (same boot ID and PID): - If this process still drives the run (heartbeat execution or native executor registered), recovery leaves the run to it (`current_controller_preparing`). - If neither drives it and the self lease lapsed, the takeover is allowed (`current_controller_released`). The existing evidence checks then choose the outcome: bootstrap again on the same run when a durable record proves no command reached the runner, or fail with a Retry (`blocked`) when runner or provider work may have started. - An unexpired self lease still waits. A claim that has not reached its executor holds it, and it lapses by itself. - A run that this process handed to the next server (graceful drain `restart_suspended` record, or the in-process hot-restart handoff marker) stays with the next server. - `heartbeat.ts`: when the executor ends with `NativeControllerDetachedForRestartError`, there was no shutdown or hot-restart handoff, and this process's own lease lapsed (`ownNativeExecutionLeaseLapsed`), heartbeat schedules restart recovery for the run when the executor exits. Recovery then reattaches the remote runner whose turn is still running, restarts an unstarted bootstrap, or fails the run with a working Retry. - `claimNativeRestartRecoveries`: a self-owned lease that is still live, with no driver here, now returns a `retryAt` at its expiry, so the wait is bounded. - `heartbeat/shutdown-handoff.ts`: add `isRunHandedOffForShutdown`. Heartbeat recovery passes it to the claim. - `suspendedByCleanDrain` now uses the shared `handedToNextControllerSinceClaim` helper. Behavior is the same. - Heartbeat test: an executor detaches after its own lease lapsed, with no restart. The same process must take the run back. Red before (timed out, the run stayed `running`), green after. - Tests: five cases for the same-boot lapsed lease: bootstrap again with no runner command; fail with a working Retry when a runner identity was recorded; leave the run to a live in-process executor; leave a handed-off run to the next server; still wait for a different live controller. ## Verification - Red before the fix: the three same-boot cases returned `awaiting_evidence` / `controller_still_alive`. - Green after the fix: - `pnpm --dir server exec vitest run src/services/native-runtime/native-runner-restart-recovery.integration.test.ts src/services/native-runtime/native-restart-recovery.test.ts src/__tests__/heartbeat-native-restart-suspension.test.ts` - Daytona wave `w-20261111T050504-s9331` from this branch (same seed and shape as s9331: 3 tasks, concurrency 3, three 90 s TM1 stalls with about 1 s between them). All 3 tasks reached `done` with no Retry. Run `632f0564` (claude) shows the exact s9331 sequence: `native.session.execute` failed after 216 s, then `reattach_remote_runner` 0.4 s later (before the third stall), then a second reattach after it, and the turn completed. Two tasks are graded FAIL only on the T1 crossing check: their own fault landed after their tool loop ended (for example loop ended 05:11:26, fault 05:11:44). Their status is `done`, with no Retry. - `tsc --noEmit -p server` is clean. ## Risks - A same-process recovery scan can now claim a run that this process recorded as its own. The guard needs all of these: the self lease lapsed, no in-process execution of the run, and no handoff to the next server. A claim that is still on its way to its executor holds a live lease, so it is not taken twice. - Upgrade safety: the decision uses only the coordinator row and in-process state. Old saved rows are judged by the same rules. A new process never matches an old boot ID, so deploys keep the existing proof path. - No new waits or file operations run before `turn.start`. - There is still no ceiling on `awaiting_evidence` for a genuinely different live controller (case c). This PR does not change that policy. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code, with tool use (shell, file edit) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude
|
fcf4e39b41 |
fix(runner): do not count stopped time against a turn start or an ACPX sidecar request (#20720)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native run starts each provider turn with `turn.start`. The control plane waits for the Runner, and an ACPX Runner waits for its sidecar, which starts the provider and sends the prompt. > - Both waits use a 30 s bound. When the host or container stops the whole tree (SIGSTOP, a host sleep), the clocks keep moving while no process can act. > - On thaw both bounds have passed. The control plane reports `PRP command turn.start timed out`, and the Runner fails the sidecar request and closes the ACPX session. > - #20704 added a stopped-time clock for startup and recovery. This pull request puts `turn.start` and its follow-up waits on the same clock, and adds the same rule to the Runner's sidecar response wait. > - The benefit is that a freeze during a turn start no longer fails the turn or closes the session. ## Linked Issues or Issue Description **What happened?** Restart-chaos wave `w-20261111T011704-s9107`, task t05 (local, claude, workload C1, fault TM2). A follow-up run sent `turn.start` at 01:22:21. The harness then stopped the server, Runner, sidecar and Claude twice: 28 s, then 33 s after a 1 s thaw. At 01:23:25 the control plane reported `provider_transport_failed: PRP command turn.start timed out`. Three same-run resumes then failed with `ACPX session is not open`, recovery stopped with `native_session_retry_exhausted`, and the task needed a manual Retry. The Retry run succeeded. **Expected behavior** Time that the processes spend stopped does not count against the turn-start bound or the sidecar request bound. A real timeout still fires while the processes run. **Steps to reproduce** 1. `cd packages/paperclip-runner && npx vitest run src/live/runnerd-codex-transport-stall.test.ts`. On master it fails with `provider_transport_failed: PRP command turn.start timed out`. 2. `cargo test --manifest-path packages/paperclip-runner/runner/Cargo.toml -p paperclip-runner-core --test acpx_sidecar_stopped_time`. On master it fails with `ACPX sidecar request timed out at initialize`. It stops the process early in the wait and again in its last half second. Note on the repeated `ACPX session is not open`: the wave ran `eafd5853d9`, which does not contain #20675. On master, #20675 settles a cut-off start from the sidecar's dispatch journal. If the prompt never left, the Runner reopens the session and sends it once. If the journal shows the prompt left, or cannot prove it, the session stays closed and the run ends with a working Retry, as the class rule requires. This pull request stops the freeze from cutting off the start in the first place. Refs #20704, #20675 ## What Changed - `packages/paperclip-runner/src/live/runnerd-codex-transport.ts`: `#startTurn` runs under `#withAdmissionClock`. The turn-start deadline, the held-tool-call wait, the start-event wait, the adoption check time and the `session.snapshot` wait all read `#waitNow()`, so they share one stopped-time credit. - `packages/paperclip-runner/runner/crates/runner-core/src/acpx_sidecar_transport.rs`: `receive_response` blocks in 1 s slices. `credit_stopped_time` treats a slice overshoot beyond 2 s as a stop. It moves the deadline by the overshoot, and to at least 5 s after the thaw, so a stop in the last slice still leaves the sidecar time to answer. Total credit per request is capped at 15 min. - The control-plane clock (`RunnerStallLedger`) now uses the same rule: a stop is credited in full plus a 5 s grace, capped at 15 min per admission. - `packages/paperclip-runner/src/live/runnerd-codex-transport-stall.test.ts` (new): drives the real transport and Runner. It SIGSTOPs the Runner tree, jumps `Date.now` by 40 s mid-wait, then resumes the Runner. It fails if the turn start leaves the clock, or if the command wait stops reading it (checked by removing `clock: this.#admissionClock`). This is the transport-level test requested in the #20704 review. - `packages/paperclip-runner/runner/crates/runner-core/tests/acpx_sidecar_stopped_time.rs` (new, its own test binary): a sidecar stops the test process for 6 s against a 3 s bound and answers after the thaw. - Unit test for `credit_stopped_time`. ## Verification - Red on master, green here: both commands in "Steps to reproduce". - `npx vitest run src/live/runnerd-codex-transport.test.ts src/live/runnerd-codex-transport-stall.test.ts src/live/runnerd-acpx-source-reconciliation.test.ts src/live/runnerd-codex-transport-adoption.test.ts`: 342/342 pass. - `cargo test -p paperclip-runner-core --lib acpx_sidecar_transport` (12 pass), `--test acpx_sidecar_transport --test acpx_park_preparation --test acpx_provider_turns` (all pass), `cargo fmt --check` clean. - `npx tsc --noEmit -p tsconfig.json` and `-p tsconfig.test.json`: clean. - Not run: a new chaos wave. ## Risks - A real stall longer than 2 s in the Runner (or 5 s in the control plane) during a turn start or a sidecar request now extends that wait by the stall plus 5 s. The waited-on process could not be observed during the stall either. The 15 min cap bounds a process that keeps stalling. - The Rust wait now wakes once per second while it waits for a sidecar reply. This is an active request with no I/O per wake. - The 1 s control-plane ticker from #20704 now also runs during a turn start, and stops when the start returns. - No persisted state, launch identity or saved-session shape changes. Old saved runs resume through the same code, and an older Runner keeps its old wait. Nothing new runs before `turn.start`, and no wait is added on the normal path. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude (Anthropic), model ID `claude-opus-5-5`, run in Claude Code with tool use (shell, file edit) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>canary/v2026.1111.0-canary.4 |
||
|
DottaandClaude
|
1fa7e92fa8 |
fix(native): restart in place a Runner that only settled an earlier attempt's suspend (#20724)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - A native run talks to a Runner process. When the control plane loses
that Runner, the run resumes itself with a limited number of same-run
attempts.
> - A server freeze while the controller suspends a finished turn can
stop the Runner inside `runner.suspend`. The controller journal then
keeps that suspend as pending.
> - The next attempt starts a new Runner. It settles the old suspend
from the welcome, records `suspended`, and exits with code 0 before it
accepts any command of the new attempt. The server reported "runnerd
exited unexpectedly with code 0", so one freeze spent one same-run
attempt. If a second fault then occurs, no attempt is left in reserve.
> - #20714 made both sides agree on that suspend's acknowledgement. This
pull request handles the clean exit that follows it.
> - During startup, the server restarts that Runner in place once,
through the existing `#recoverRunnerProcess` path, and only when the
durable Runner journal proves the exit.
> - The benefit is that one freeze during a suspend costs zero same-run
attempts.
## Linked Issues or Issue Description
Refs #20714.
**What happened?**
A restart-chaos task (local Runner, Claude, a freeze of the server
during a live turn) needed a manual Retry. #20714 fixed the identity
mismatch. The second same-run attempt still ended with
`runner_local_connect_failed: runnerd exited unexpectedly with code 0`,
because that Runner only settled the suspend left by the first attempt
and then exited as suspended.
**Expected behavior**
The attempt continues. The server restarts the Runner once. The new
Runner starts from a clean `suspended` state and serves the attempt's
commands. No provider work is repeated.
**Steps to reproduce**
1. Start a native run. Let the controller send `runner.suspend` when it
closes the session.
2. Stop the Runner after it journaled the suspend and before the
controller records the result (a server freeze does this).
3. Resume the same run. The new Runner settles the old suspend and exits
with code 0, and startup fails.
The new live test reproduces this with the real runnerd and the fake
Codex provider.
**Paperclip version or commit**
`master` at
|
||
|
DottaandClaude
|
e039be9202 |
fix(runner): replace a follow-up session whose remote workspace root moved (#20712)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs keep an execution lease. Restart recovery takes over a run only when it can prove that the recorded controller stopped. > - A stall (SIGSTOP) longer than the lease, with no restart, makes the executor lose its own lease. The executor then detaches for "the next controller" and exits. > - The next controller is this same process. Recovery asks the PID probe if the recorded controller is alive. The recorded controller is this process, so the answer is always "alive". > - The task then waits forever: `in_progress`, no Retry, an active run, and an active environment lease. > - This pull request lets recovery recognise its own process. When this process no longer drives the run and its own lease lapsed, the prior owner is gone. The existing evidence checks then decide, as for any stopped controller. > - The benefit is that a same-boot stall never needs a manual Retry. When provider work may have started, the run fails with a working Retry instead of a wait with no end. ## Linked Issues or Issue Description Refs #20694 (the same class of fix for legacy `preparing` runs), #20529, #20691. **What happened?** Chaos wave `w-20261111T034105-s9330`, task t02 (Daytona, OpenCode, fault TM1). The server process was frozen with SIGSTOP for about 84 s while run `89a3922b` prepared. After SIGCONT, `native.process_start_requested` was recorded and `native.session.execute` failed after 118 ms: the lease renewal found the lease lapsed, and the executor detached and exited. From then on, recovery recorded `native.recovery.transition` "waiting for safe ownership evidence" with reason `controller_still_alive` every 30 s for more than 20 minutes. The run stayed `running`, the issue stayed `in_progress`, no Retry was shown, and the environment lease stayed `active`. **Expected behavior** When this process no longer drives a run that it recorded as its own, recovery takes the run back. No turn reached the runner here, so the bootstrap restarts on the same run. When a turn or runner work may have started, the run fails and shows a working Retry. **Steps to reproduce** 1. Start a native remote (Daytona) run. 2. Send SIGSTOP to the server during preparation, and hold it longer than the execution lease. 3. Send SIGCONT. The executor loses its lease, detaches and exits. 4. See that every recovery scan waits with `controller_still_alive`, and the run never settles. The new tests in `native-runner-restart-recovery.integration.test.ts` ("same-boot controller whose own lease lapsed") reproduce this at the recovery layer. ## What Changed - `claimNativeRestartRecoveries`: when the recorded controller is this exact process (same boot ID and PID): - If this process still drives the run (heartbeat execution or native executor registered), recovery leaves the run to it (`current_controller_preparing`). - If neither drives it and the self lease lapsed, the takeover is allowed (`current_controller_released`). The existing evidence checks then choose the outcome: bootstrap again on the same run when a durable record proves no command reached the runner, or fail with a Retry (`blocked`) when runner or provider work may have started. - An unexpired self lease still waits. A claim that has not reached its executor holds it, and it lapses by itself. - A run that this process handed to the next server (graceful drain `restart_suspended` record, or the in-process hot-restart handoff marker) stays with the next server. - `heartbeat/shutdown-handoff.ts`: add `isRunHandedOffForShutdown`. Heartbeat recovery passes it to the claim. - `suspendedByCleanDrain` now uses the shared `handedToNextControllerSinceClaim` helper. Behavior is the same. - Tests: five cases for the same-boot lapsed lease: bootstrap again with no runner command; fail with a working Retry when a runner identity was recorded; leave the run to a live in-process executor; leave a handed-off run to the next server; still wait for a different live controller. ## Verification - Red before the fix: the three same-boot cases returned `awaiting_evidence` / `controller_still_alive`. - Green after the fix: - `pnpm --dir server exec vitest run src/services/native-runtime/native-runner-restart-recovery.integration.test.ts src/services/native-runtime/native-restart-recovery.test.ts src/__tests__/heartbeat-native-restart-suspension.test.ts` - `pnpm verify` (see the PR comments for the selected scope). - `tsc --noEmit -p server` is clean. ## Risks - A same-process recovery scan can now claim a run that this process recorded as its own. The guard needs all of these: the self lease lapsed, no in-process execution of the run, and no handoff to the next server. A claim that is still on its way to its executor holds a live lease, so it is not taken twice. - Upgrade safety: the decision uses only the coordinator row and in-process state. Old saved rows are judged by the same rules. A new process never matches an old boot ID, so deploys keep the existing proof path. - No new waits or file operations run before `turn.start`. - There is still no ceiling on `awaiting_evidence` for a genuinely different live controller (case c). This PR does not change that policy. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code, with tool use (shell, file edit) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude
|
e719d42557 |
fix(runner): accept the acknowledgement of an indeterminate suspend replay (#20714)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native run talks to a Runner process (runnerd). The controller in the server sends commands. The Runner journals each command before it acts and replies with a result. > - `runner.suspend` and `runner.shutdown` are terminal commands. The Runner waits for a `command_result_ack` before it exits. It compares that acknowledgement with the receipt that it journaled. > - A Runner that is killed inside `runner.suspend` journals the command but not its result. The next Runner replays the command as `indeterminate`. It sends it on the wire as `failed` with `execution_indeterminate`, and it still treats it as terminal. > - The controller acknowledged the `failed` status that it received. The Runner compared that with its journal status `indeterminate` and exited with "terminal command result acknowledgement changed its durable identity". The controller also lifted its terminal fence for a failed suspend and sent the next command to a Runner that was about to exit. > - This pull request makes both sides agree on the status of that one receipt, and keeps the fence. > - The benefit is that a freeze or restart during a suspend no longer burns the same-run retries and leaves the task at a manual Retry. ## Linked Issues or Issue Description **What happened?** A restart-chaos task (local Runner, Claude, a 28-second freeze of the server during a live turn) needed a manual Retry. Retry worked. 1. The turn completed. The server then sent `runner.suspend`. The server froze before the Runner reported the result. 2. The close found the Runner not durably suspended (`commandStatus: pending`, `runnerLifecycle: ready`). The run failed with `native_runner_checkpoint_owner_lost`, and same-run recovery stopped the old runnerd. The suspend was journaled but not finished. 3. Attempt 2 started a new runnerd on the same state. The welcome delivered the stale suspend. The Runner replayed it as `indeterminate` and sent `failed`. The controller acknowledged `failed`. The Runner exited with code 1: "terminal command result acknowledgement changed its durable identity". The controller had recorded the suspend as `failed`. 4. Attempt 3 found the fence for a suspend that the controller no longer listed as pending. The Runner reconciled it and exited with code 0. The server reported "runnerd exited unexpectedly with code 0", and `native_session_retry_exhausted` stopped automatic recovery. **Expected behavior** The controller acknowledges exactly the receipt that the Runner sent. The Runner accepts that acknowledgement, records `suspended`, and the next attempt starts from a clean suspended Runner. **Steps to reproduce** 1. Start a native run with a local Runner. Let the turn complete so that the server sends `runner.suspend`. 2. Freeze the server (SIGSTOP) for about 30 seconds before the suspend result arrives, so that recovery stops the Runner inside the suspend. 3. Let same-run recovery start a new Runner. It exits with "terminal command result acknowledgement changed its durable identity". The unit tests in this PR reproduce the same sequence without a live run. **Paperclip version or commit** `master` at |
||
|
DottaandClaude
|
156212cfd5 |
test(chaos): close a covering modal before clicking Retry, and name it when it blocks (#20715)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart chaos harness (`tests/restart-chaos/`) clicks every
needed Retry in the real board UI with Playwright.
> - In one wave, a task failed and showed Retry. The "You completed 5
tasks!" star modal was open over the page.
> - The modal is a Radix dialog. It sets `aria-hidden="true"` on the
rest of the page, so `getByRole('button', { name: 'Retry' })` did not
find the visible Retry.
> - The grade said `ui_retry_button_missing`. That name hides the real
cause.
> - This pull request makes the harness close a covering modal before it
looks for Retry. When a modal does not close and keeps a visible Retry
unreachable, the grade says `ui_retry_blocked_by_modal`.
> - The benefit is that a chaos grade names the real cause, and a
promotional prompt does not fail a wave.
## Linked Issues or Issue Description
Refs #20713 (product fix: the star prompt no longer opens over a task
that needs Retry)
**What happened?**
A restart chaos wave saved a "Retry missing" screenshot. The screenshot
shows the Retry banner and the star modal over it. The saved DOM shows
the Retry button as visible, with a box, inside the banner. The harness
still graded the task `ui_retry_button_missing`.
**Expected behavior**
The harness closes a modal that covers the page, as a user does, and
then clicks Retry. If a modal does not close, the grade names the modal
as the cause.
**Steps to reproduce**
1. Open a task page that shows a failed attempt with Retry.
2. Open a Radix modal over the page (the star prompt did this).
3. Run `BoardUi.clickRetry` on that task. It reports
`ui_retry_button_missing`.
## What Changed
- `board-ui.ts`: before and while it waits for Retry, `clickRetryOnPage`
closes an open modal that does not itself offer Retry. It clicks "Not
now" (or "No thanks", "Maybe later", "Dismiss") when present, else it
presses Escape. It also closes a modal that opens after Retry appears
and swallows the click, then clicks again. At most three dialogs are
closed. Each click has a timeout equal to the per-button request wait.
- The click result has a new optional `dismissedDialogs` list (first
line of each closed dialog). The observer copies it into the task's
retry attempt.
- New failure reason `ui_retry_blocked_by_modal` (in `types.ts`): a
visible Retry exists in the DOM, it is not reachable by role, and a
modal stays open. R3 in `grader.ts` reports it with the banner text,
like the other UI reasons.
- `README.md`: the board UI Retry paragraph names the dismissal and the
new reason.
- Tests in `board-ui.test.ts` use real Chromium pages with a modal and
an `aria-hidden` page: "Not now" closes it and Retry is clicked; Escape
closes it and Retry is clicked; a modal that opens on hover over Retry
is closed and Retry is clicked; a modal that does not close gives
`ui_retry_blocked_by_modal`. `grader.test.ts` covers the new R3 detail.
## Verification
- Red first: the three new `board-ui.test.ts` cases failed on master
(the covered Retry was reported as missing).
- Green: `cd tests/restart-chaos && npx vitest run board-ui.test.ts
grader.test.ts triggers.test.ts` (with `observer.test.ts`) gives 99
passed.
- `npx tsc --noEmit -p tests/restart-chaos/tsconfig.json` passes.
- `pnpm verify` exits 2 because the README is a global CI input. CI
supplies the broad coverage.
## Risks
- Low risk. This changes only the test harness.
- The harness can close a dialog that the product meant to keep open.
The closed dialog is recorded in `dismissedDialogs`, so the evidence
still shows it.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), in Claude Code with
tool use (shell, file edit, test runs).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude <noreply@anthropic.com>
|
||
|
DottaandClaude
|
f77b8a288e |
fix(ui): keep the GitHub star prompt off tasks that need a Retry (#20713)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A task that fails shows a banner with a Retry button. The user
clicks Retry to run the task again.
> - Self-hosted boards also show a "You completed 5 tasks! Star on
GitHub" modal (#20453).
> - The modal can open on the task page while the task shows its Retry
banner.
> - The modal is a Radix dialog. It sets `aria-hidden="true"` on the
rest of the page and sets `pointer-events: none` on the body. The Retry
button is visible, but a click does not reach it and assistive
technology cannot find it.
> - This pull request makes the star prompt wait for a calm moment. It
does not open on a task or agent chat page, and it does not open while a
task Retry or status banner is on screen.
> - The benefit is that a visible Retry is always reachable.
## Linked Issues or Issue Description
Refs #20453
**What happened?**
A restart chaos run left a task in the failed state with the Retry
banner. The "You completed 5 tasks!" star modal opened over the same
page. The Retry button was in the DOM and visible. Playwright
`getByRole('button', { name: 'Retry' })` did not find it, because the
modal had set `aria-hidden="true"` on the banner. The body also had
`pointer-events: none`, so a user click on Retry did not reach the
button.
**Expected behavior**
A promotional prompt never covers a task that needs the user. A visible
Retry is always reachable.
**Steps to reproduce**
1. On a self-hosted instance, finish five tasks in one company. Do not
open the star prompt before.
2. Let a sixth task fail so that it shows "Couldn't finish this attempt"
with Retry.
3. Open that task. The star modal opens over the Retry banner.
**Paperclip version or commit**
master at
|
||
|
DottaandClaude
|
08ea18085e |
fix(workspaces): restore a saved workspace into one copy of sandbox disk (#20695)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A task on a remote (Daytona) sandbox continues from its saved workspace. Before the agent starts, the server uploads the saved archive into the sandbox and a sandbox script extracts it. > - The extraction wrote a full second copy of the workspace beside the uploaded archive. A 7.5 GB workspace on a fresh sandbox then needed about 15 GB, and the restore failed with `stage=extract code=ENOSPC disk=full`. > - The failure was classified as retryable, so every automatic retry and every Retry repeated it on a same-size sandbox. The task could not continue. > - On a retained sandbox it is worse: the old root, the archive, and the new tree all share the disk. > - This pull request makes the restore need about one copy of the workspace. The sandbox removes only data that control-plane storage already holds durably, and it never removes a root whose save the server did not prove. > - The benefit is that a workspace that fits once on the sandbox disk restores. A disk that is too small for one copy now gets a named failure that does not loop. ## Linked Issues or Issue Description **What happened?** A saved remote workspace of about 7.5 GB could not be restored on a new Daytona sandbox. The run failed with `remote_workspace_restore_failed: step=sandbox_restore; remote_workspace_operation_failed: operation=restore exit=1; remote_workspace_restore_failed: stage=extract code=ENOSPC disk=full exit=1`. The run marked the failure as retryable, and each retry failed the same way. **Steps to reproduce** 1. Save a remote workspace whose archive is larger than half of the free sandbox disk. 2. Start the next run on another sandbox, so that the server restores the saved archive. 3. The upload succeeds. The extraction writes the second copy and fails with ENOSPC. **Expected behavior** A workspace that fits once on the sandbox disk restores. A disk that cannot hold one copy gives a specific failure that is not retried automatically, and Retry works after the sandbox has more space. **What the disk holds during a restore (from the code)** - The uploaded archive (`<parent>/.paperclip-runtime/workspace-snapshots/<operation>/workspace.pcws`, about the workspace size, because the archive is not compressed). - The extracted tree (`<root>.restoring-*`, the workspace size). - On a retained sandbox only: the old root, which stayed until after the new root was published. - After a restore that died: its partial tree and its archive, which no later attempt removed. Refs #20678 (the save side of the same full-disk class). ## What Changed - `workspaceArchive` restore has a `consumeArchive` mode. It creates the directories and symlinks first, then writes the file bodies from the last to the first. After it verifies each file against the manifest hash, it truncates the archive behind that file. The archive and the tree together need the workspace size plus the largest file. The default mode, which the server uses for local restores, does not change. - The sandbox restore script uses that mode on a private rename of the uploaded archive. A later upload to the same path cannot change the file that is being consumed. The script removes the consumed file when it stops. - The helper's own staging is now inside the root-specific `<root>.restoring-*` folder, not in a shared `.workspace-restore-*` sibling. - While it holds the root's restore lock, and after the epoch fence, the script removes leftovers of an earlier attempt on the same root that died: `<root>.restoring-*` folders, `.consuming` archives in its own staging, and archive staging folders that a `restore-target.json` binds to the same root. Each restore writes that binding after the reclaim step. - When the server proved that a retained root is its own earlier, fully saved allocation (`replaceRetiredRoot` and a Paperclip receipt, as before), the script removes the root's files before the extraction, not after the publish. The receipt and `.paperclip-runtime` stay, so an interrupted restore proves the same root again and continues. - A full disk at any restore stage reports `reason=sandbox_disk_full`. The existing named-refusal rule makes it non-retryable, so it goes to the board recovery action and does not loop. - `doc/execution-semantics.md` describes the disk need, the reclaim, the early removal, and the new reason. ## Verification - `cd server && npx vitest run src/services/remote-workspace-transport.test.ts src/services/remote-workspace-snapshot-archive.test.ts` (287 passed). - `cd server && npx vitest run src/services/remote-workspaces.integration.test.ts src/services/remote-workspace-lifecycle.test.ts src/services/remote-workspace-archive-worker.test.ts src/services/remote-workspace-snapshot-archive-environments.test.ts` (122 passed, 16 skipped). - `cd server && npx tsc --noEmit -p tsconfig.json` (pass). - The new tests run the real sandbox scripts against a folder with a byte capacity. A `--require` preload makes each file write fail with ENOSPC when the folder would exceed the capacity. - Red on `origin/master`, green with this change: - Fresh sandbox, room for the archive plus one file: master fails with the exact production error `stage=extract code=ENOSPC disk=full exit=1`. The fix restores every byte and leaves only the restored files. - Retained root that the server proved, room for the old root, the archive, and one file: master fails with ENOSPC. The fix restores and keeps the runner state. - Too small for one copy: the failure is `stage=extract reason=sandbox_disk_full`, with no partial tree or archive left. A retry with more space then restores. - Both sandbox processes are killed mid-extraction (a restart): a retry with the same identity and a retry with a new identity and epoch both reclaim the leftovers and restore into the same capacity. - The command dies right after it removed a proven root's files: the next restore proves the root again from its kept receipt, restores, and keeps the runner state. - Integration test through `prepareRemoteWorkspace`: a full disk gives `retryable: false` with `reason=sandbox_disk_full`. On master the same case is `retryable: true`. - Guards (pass on master and with the fix): a retained root without the server proof, or without a Paperclip receipt, is refused with `reason=occupied_root`, and none of its files are removed. - Previous-release story: the archive format and the saved state do not change. A partial `<root>.restoring-*` tree from the previous release is reclaimed (test). Archive staging from the previous release has no binding, so it is kept (test). The previous release's shared `.workspace-restore-*` siblings cannot be bound to a root, so they are kept. ## Risks - The consumed archive cannot be read again. Each restore uploads a new copy from control-plane storage, and the early-return path for an already restored root does not read the archive. Every file is still checked against the manifest, and the manifest is checked against the saved digest. - The reclaim step removes data only under the root's restore lock and after the epoch fence. It removes only names that a restore of the same root creates. Staging that names another root, or no root, is kept. - The early removal of a retained root depends on the existing server proof (`isRetiredOwnAllocation`) and the receipt check. If a restore fails after the removal, the saved version in storage is the source of truth, and the next restore proves the same root again. - Files in a lower overlay layer of a resumed sandbox image do not free space when they are removed. Such a sandbox can still be full. - A workspace that is larger than the free disk after the upload, plus its largest file, still fails. It now fails with the named, non-retryable `reason=sandbox_disk_full`. - No change to `packages/paperclip-runner`, the database schema, or the API. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude
|
855e241887 |
fix(server): record a long event-loop block with the deltas that name its cause (#20709)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - One server process owns the controller leases of its runs, renews them on timers, and serves every request > - On a hosted instance, the process stopped for about 103 seconds without a restart. Every lease lapsed, and two runs needed a manual Retry (the symptom is fixed in #20694) > - Nobody can say what blocked the process. The server records no event-loop delay, and it has no CPU, GC or memory record for that minute > - This pull request logs one local line for each event-loop block of 2 s or more. The line holds the deltas that tell the possible causes apart, and a likely-cause class > - The benefit is that the next stall names its class (main-thread CPU, GC, paging, CPU throttling, host suspend, or a blocking sync call) and its exact time, so we can match it to the run timeline and fix the cause ## Linked Issues or Issue Description Refs #20694 **What happened?** On a hosted instance, the merged run-event timeline of all active runs had no events from 08:19:52.04Z to 08:21:35.37Z (103.3 s). Events from streaming runs arrived about every 0.5 s before and after the gap. The process did not restart (the run diagnostic shows an observer uptime of about 24 minutes when the reaper ran). The first event after the gap was a span with a 60 s timeout that ended after 103.5 s. Thus the timer callback ran about 43 s late. This agrees with a blocked event loop, or with a frozen process. Nothing in the process recorded which one. **Expected behavior** A long event-loop block is a product failure. It lapses every lease and freezes every user. The server must record each long block with enough data to identify its cause. **Steps to reproduce** 1. Start the server. 2. Block the main thread for more than 2 s, for example with a synchronous read on a slow disk or a busy loop. 3. Before this change: the logs show nothing. After this change: the logs show `event loop blocked for <n> ms (likely <cause>)` with the measured deltas. ## What Changed - `server/src/event-loop-stall-monitor.ts` (new): one unref'd 1.5 s timeout, re-armed after each check, measures how late it runs. (Server startup creates no interval timers, so it is not a `setInterval`.) When a check is 2 s or more late, it logs once with these fields: - stall duration, start and end time (wall clock and monotonic clock) - process CPU (user and system), GC pause time and count (`perf_hooks` GC entries) - major and minor page faults, file-system reads and writes, voluntary and involuntary context switches (`process.resourceUsage()`) - cgroup v2 `throttled_usec` delta (Linux; read asynchronously, best effort) - RSS, heap and external memory - `likelyCause`: `host_suspended`, `gc`, `memory_paging`, `cpu_throttled`, `main_thread_cpu`, or `blocked_off_cpu` (sync file I/O, a sync child process, or another off-CPU wait) - `server/src/index.ts`: starts the monitor when the listener binds. It stops the monitor at the start of shutdown, and when a failed startup closes the listener. It writes the line with the existing `logger.warn`. - This is a local log line only. It is not the Telemetry path and not the Observability (OpenTelemetry) path. No data leaves the process. ## Verification - `pnpm --filter @paperclipai/server exec vitest run src/event-loop-stall-monitor.test.ts src/__tests__/server-startup-feedback-export.test.ts` (131 passed; the monitor file passed 5 runs in a row): - an 800 ms busy loop is recorded as `main_thread_cpu`, with the stall and CPU time - an 800 ms `Atomics.wait` (blocks without CPU, like a sync read on a stalled disk) is recorded as `blocked_off_cpu` - nothing is logged while the loop is responsive, or after `stop()` (injected clock) - the file-read delta of a stall is kept while the cgroup read is pending - server startup still creates no interval timer - a 100 s stall with 90 s of cgroup throttling is recorded as `cpu_throttled`, with the stall start and end times - each cause class is selected from its deltas - `pnpm --filter @paperclipai/server exec tsc --noEmit` passes. - CI supplies the broad server suites. ## Risks - Low risk. The timeout reads only process counters. Every 10 s it reads one small cgroup file asynchronously. It does not query the database or the network, so it does not keep an idle instance or its database awake. It is unref'd and stopped on shutdown. - `process.cpuUsage()` counts all threads of the process, not only the main thread. A busy libuv thread pool during a stall can make an off-CPU block look like `main_thread_cpu`. The raw deltas are in the line, so a reader can check this. - If a platform suspends and resumes the process, each resume can log one `host_suspended` line. This is expected and useful. - The monitor names the class of a block, not the call stack. If the class is not enough, the next step is a bounded CPU profile around the stall time. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude Code with tool use, shell, and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes (the module comment carries the contract) - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude
|
6d1eae73c0 |
fix(runner): do not count time the control plane was stopped against a Runner command wait (#20704)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The native runtime resumes a run after a control-plane restart. It launches a Runner and waits for recovery barriers (`run.attach`, a `runner.drain` probe, the provider identity). > - These waits poll with a wall-clock deadline. When the control plane and its Runner are stopped together (SIGSTOP, host sleep), the wall clock passes the deadline before either process can act. > - On thaw, the wait reports a timeout although nothing was slow. Recovery then spends an attempt and a retry delay. > - This pull request gives each startup or recovery admission a clock that does not count stopped time. > - The benefit is that a freeze or host sleep during an admission no longer turns into a false `PRP command ... timed out` failure. ## Linked Issues or Issue Description **What happened?** A restart-chaos wave (local, codex, fault TM2) stopped the server and its Runner tree for 44 s, 1 s after a resumed Runner started. The resume barrier `runner.drain` had a 30 s deadline. On thaw, the server reported `runner_local_connect_failed: PRP command runner.drain timed out` 45 s after the wait began (`provider.session.load` span 45130 ms). Recovery spent attempt 1 and waited 30 s more before attempt 2. **Expected behavior** Time that the waiting process spends stopped does not count as waiting time. A real timeout still fires when the Runner does not answer while both processes run. **Steps to reproduce** 1. Start a Runner command wait with a 30 s deadline. 2. After 1 s, stop the process for 44 s (the test moves the system clock without running timers). 3. Let the Runner complete the command 2 s after thaw. 4. Before this change, the wait rejects with `runner.drain timed out`. After this change, it resolves. Note: in that chaos task, this false timeout was not the reason that the task needed a manual Retry. An earlier SIGKILL of the whole server tree stopped the Codex turn while a foreground shell command (`./bench.sh`) was running. On restore, Codex had no active turn (`provider_turn_lost_on_restore`). Recovery correctly blocked automatic replay (`uncertain_provider_action`), because no durable record proves that the interrupted command had no side effect. The manual Retry then completed the task. This PR does not change that policy. Refs #20675 ## What Changed - `packages/paperclip-runner/src/live/runnerd-codex-transport.ts`: add `RunnerStallLedger`, the clock of one startup or recovery admission. `now()` is wall time minus the stopped time observed since the admission began. Its poll loops and a 1 s ticker share one last-observation time, so each stop is credited once, wherever the process was when it stopped (a poll loop, a checkpoint flush, a spawn). Only the part of a gap beyond 5 s is credited. - `#start` and `#resume` run under one ledger (`#withAdmissionClock`). A nested startup keeps the outer ledger. The ticker stops when the admission ends. - During an admission, `waitForRunnerCommand`, the provider-identity wait, the cold admission deadline, the recovery admission deadline, and the default 30 s command deadline all use this clock. - Waits after admission, including `turn.start` and its event and snapshot waits, keep wall-clock deadlines. Their behavior does not change. - Tests in `runnerd-codex-transport.test.ts`: the chaos sequence (red before, green after), the unchanged wall-clock timeout outside an admission, a barrier that still times out after a freeze, a 70 s freeze between two barriers on one cold deadline, and a slow-but-running process that still reaches its deadline. ## Verification - Red: `cd packages/paperclip-runner && npx vitest run src/live/runnerd-codex-transport.test.ts -t "stopped"` failed before the fix with `runner_local_connect_failed: PRP command runner.drain timed out`. - Green: the same command passes after the fix. - `npx vitest run src/live/runnerd-codex-transport.test.ts`: 301/301 pass, including the existing "shares Pi's absolute cold deadline across recovery barriers and rejects a late ACK" test. - `npx tsc --noEmit -p packages/paperclip-runner/tsconfig.json`: clean. - `pnpm verify` exits 2 (changes outside resolved test graphs). Required CI supplies the broad coverage. - Not run: a new chaos wave. The unit tests reproduce the exact wait and clock sequence from the artifacts. ## Risks - A real event-loop hang longer than 5 s during an admission now extends that admission by the hang minus 5 s. The Runner answer could not be observed during the hang either. Each credited gap still costs 5 s of budget, so a process that is slow on every observation still reaches its deadline. - The ticker runs only while a startup or recovery admission runs (an active operation), every 1 s, does no I/O, and is cleared in `finally`. It does not keep an idle process awake. - A `turn.start` that is cut by a freeze still reports a wall-clock timeout, as before. That case needs its own change, because its event and snapshot waits share one deadline. - No persisted state, launch identity, or saved-session shape changes. Old saved runs resume through the same code. A deploy mid-run only changes how long an in-process admission wait tolerates a stop. No new wait or file operation runs before `turn.start`. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude (Anthropic), model ID `claude-opus-5-5`, run in Claude Code with tool use (shell, file edit) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude
|
200c6e40e2 |
fix(native): restart a bootstrap whose Runner suspended before the provider opened (#20691)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - When a native run's first provider start fails before any session opens, Paperclip retries the bootstrap on the same run. Durable controller evidence proves that no provider session or provider event exists. > - A remote sandbox keeps the Runner state inside the workspace. A replacement sandbox gets that state back when the saved workspace is restored. > - When the provider refused `session.open` (for example an Anthropic 429), the Runner still suspended cleanly. Its state says `suspended`, but the provider state has no session identity. > - The same-run restart handled a Runner that never suspended. It did not handle a Runner that suspended before the provider opened a session. It failed with `runner_harness_state_mismatch: provider_identity_incomplete (provider_not_suspended)`, and the task showed "Recovery needed" without Retry. > - This pull request lets that proven restart set the unopened session aside and start the provider again. > - The benefit is that a rate-limited provider start continues by itself after the limit resets. ## Linked Issues or Issue Description Refs #20627 and #20683 (other restore paths). This PR changes only the same-run bootstrap restart. **What happened?** Two ACPX Claude tasks on remote Daytona sandboxes, on staging: 1. The first attempt sent `session.open`. ACPX rejected it with `classification=provider_rate_limit` (Anthropic 429). The Runner exited with `lifecycle=suspended`. 2. Paperclip recorded `recoveryMode: bootstrap_retry`, `providerSessionEstablished: false`, `providerEventsExist: false`, and a retry after the limit reset. 3. One hour later the retry ran as `bootstrap_incomplete` on the same run, in a new sandbox with the saved workspace restored. 4. Host preparation inspected the restored Runner state. The Runner was `suspended`, but the provider had no session identity. The inspection threw `provider_identity_incomplete (provider_not_suspended)` before the bootstrap restart branch ran. 5. The run failed. The task stopped at "Recovery needed" with only inspect actions. **Expected behavior** The restart sets the unopened session aside and opens a new provider session, as it already does for a Runner that never suspended. **Steps to reproduce** 1. Restart a remote run as `bootstrap_incomplete` with a proven bootstrap restart. 2. Give the sandbox a Runner state for this run with `lifecycle: suspended` and a provider state with no session identity. 3. Prepare the host. On `master` it throws `runner_harness_state_mismatch: provider_identity_incomplete`. The new `interrupted_bootstrap_unopened` cases of `bootstraps only an untouched provider session in a $outcome workspace lease` run these steps for a created and a resumed lease. **Paperclip version or commit** `master` at `7435abcc63`. ## What Changed - During host preparation, a Runner state that is suspended with no provider session identity returns the new incomplete reason `provider_unopened`, but only when all of these hold: - the run is restarting as `bootstrap_incomplete`; - the state belongs to this same run; - the controller already recorded the bootstrap restart proof (its durable journal shows no provider turn, and the old Runner is stopped or absent). - the provider state is the one saved before any session opened (`remoteProviderStateNeverOpened`): `prepared`, no session or identity, no turn in progress, no unconfirmed exit. Only ACPX and Codex-family states qualify. - The created-lease and resumed-lease restart branches share one check (`restartableBootstrapState`) and accept `provider_unopened` exactly like `not_suspended`. They move the old session aside and claim a new, untouched session root. - Every other case still throws `provider_identity_incomplete`: another run's state, a restart without the proof, a normal resume, and close checkpointing. ## Verification - `pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/native-session-executor.test.ts -t "bootstraps only an untouched provider session"`: 47 passed. - Red on `master`: the `created` and `resumed` `interrupted_bootstrap_unopened` cases fail with `runner_harness_state_mismatch: provider_identity_incomplete (identity_binding)`. Green with this change. - Fail-closed cases stay red-proof: `interrupted_bootstrap_unopened_unproven` (no restart proof) and `interrupted_bootstrap_unopened_other_run` (another run's state) still throw. - Full `native-session-executor.test.ts` and `pnpm --filter @paperclipai/server typecheck` pass. ## Risks - Low risk. The new branch needs the same proof that already authorizes a fresh bootstrap for a Runner that never suspended. The proof comes from the controller's durable journal, not from the sandbox. - The normal path is unchanged and has no added wait. - Not proven live: I did not confirm on staging that the controller recorded the restart proof for these runs. If it did not, the run still fails closed as before. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude
|
4baff83225 |
fix(chat): never batch messages of two chat conversations into one wake (#20697)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Chat channels let a linked user talk to an agent from GitHub, Slack and other platforms. Each external thread is a chat conversation. Many conversations can share one task. > - A reviewed chat run attests exactly one conversation. #17054 keeps each conversation in its own wake batch, but it compares only the conversation that a wake records. > - The work-intent outbox saves unanswered user messages as a plain comment wake. That wake records no conversation, even when its messages are chat messages. A wake with no recorded conversation matched every conversation, so a message from another conversation merged into it. > - The promoted run held messages of two conversations. The binding check correctly refused it (`reviewed_chat_execution_binding_not_authorized`, `binding_denied`), and the task needed Retry. > - This pull request compares the conversations of the queued messages themselves. Wakes of different conversations never merge. Each conversation gets its own run, in order. > - The benefit is that replies in several threads of one task each get an attested run, without a failed run or a Retry. ## Linked Issues or Issue Description Refs #17054 (the earlier per-conversation batching fix). Refs #20689 (a different shape of the same error). **What happened?** On staging, a GitHub pull-request review task had two runs fail with `setup_failed` / `reviewed_chat_execution_binding_not_authorized` (`reviewedChatBinding.denialReason: binding_denied`). Each failed run held the same two older saved messages plus one new message. The two runs recorded two different `chatConversationId` values. The work-intent outbox had saved the older messages as plain deferred wakes with no conversation. A new message from each conversation then merged into one of these saved wakes. **Steps to reproduce** 1. Bind two chat conversations to one `in_review` task. 2. Let the outbox save an unanswered message of conversation B as a deferred wake while a run is active. 3. As the linked user, send a message in conversation A while the run is still active. 4. See that the message of A merges into the saved wake of B. The promoted run fails the binding check. **Expected behavior** The saved message of B runs as its own wake. The message of A runs after it, in its own attested run. ## What Changed - New `server/src/modules/wake-queue/adapters/chat-conversation-batch.ts`: reads the inbound chat links (`chat_message_links`, direction `inbound`) of queued messages and finds messages of a conversation other than the expected one. - Deferred-wake lookup (`findExistingDeferredWake`): when the incoming wake has a conversation, a saved wake qualifies only if none of its queued messages is an inbound message of another conversation. This is one more predicate in the same query (no new round trip). It uses the existing `chat_message_links_company_comment_idx` index. - Coalescing into the active run (`coalesceIntoActiveExecutionRun`): returns no merge when the merged batch would hold messages of a conversation other than the one it records. The wake is then saved behind the run, which is the existing fallback. - Release drain (`findNextDeferredWake`), upgrade repair: a saved chat wake that older code already merged across two conversations is split. The other conversation's messages move to a plain saved wake (the outbox shape), ordered just before it. The original wake keeps its conversation and its own messages. The drain then reads the queue again. - `doc/execution-semantics.md`: describe the comparison and the upgrade repair. - New test `chat-wake-conversation-batch.test.ts` (embedded Postgres, real admission and release code, real `authorizeChatConversationForBoundRun`). ## Verification - Red on `origin/master` (test file only): `pnpm --dir server exec vitest run src/modules/wake-queue/adapters/chat-wake-conversation-batch.test.ts` gives 4 failed, 6 passed. The 4 failures are the bug: the new message merges into the other conversation's saved wake (`[[earlier, reply]]`), two conversations cross-merge, a message coalesces into a queued run of another conversation, and the old merged shape stays merged. The 6 passing cases pin unchanged batching. - Green with the fix: 10 of 10 pass. Each separated wake passes `authorizeChatConversationForBoundRun` for its own conversation. The old merged shape fails it with `paperclip_runner_chat_attachment_binding_denied` before the split. - Batching that stays the same (pinned, pass before and after): comment wakes with no conversation merge; messages of one conversation merge; an outbox-saved wake of the same conversation still absorbs the reply; a chat message and plain comments merge in both orders; same-conversation messages still coalesce into a queued run; two recorded conversations stay apart. - Related suites pass locally: `src/modules/wake-queue`, `src/modules/run-dispatch`, `chat-run-milestone-work`, `durable-chat-wakeup`, `heartbeat-comment-wake-batching`, `heartbeat-reviewed-chat-binding.integration`, `slack-conversation-lifecycle`, `queued-comment-consumption`, `issue-queued-comments-routes` (23 files, 1037 tests). `heartbeat-process-recovery`: see the comment below. - `tsc --noEmit -p server/tsconfig.json` passes. ## Risks - Low risk. No schema change and no migration. - Behavior change only for chat messages: a message no longer joins a saved wake or a run that holds another conversation's inbound messages. It gets its own wake behind them, and it runs after them. Before this change, that batch always failed the binding check. - Board comments have no inbound chat link, so they belong to no conversation. Wakes without a conversation, wakes of one conversation, and mixed board-comment and chat wakes batch exactly as before. - Ordering: the deferred queue still promotes one wake per release, oldest first. The second conversation waits for the issue execution lock. Single-assignee and atomic checkout are unchanged. - Upgrade: a saved wake that older code merged across conversations is split at promotion. No message is dropped or delivered twice: each message id moves to exactly one wake. A run that older code already promoted in the merged shape is not changed; it fails the binding check as before and keeps its existing Retry path. - No wait is added before a turn starts. Admission adds one predicate to an existing query. Coalescing adds one indexed read only when the merged batch has a chat conversation. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), in Claude Code with tool use (shell, file edits, GitHub CLI). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>canary/v2026.1111.0-canary.3 |
||
|
DottaandClaude Opus 5.5
|
d79b7f29f2 |
fix(heartbeat): bind queued card-answer interrupts to the delivered answer (#20698)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A person can queue direction while a task's run is active, then use Interrupt to stop the run and start a successor that receives the direction > - The queued direction can be an answer to a card, for example an accepted confirmation of a saved plan > - `explicitOperatorRunIdentity` verifies the click from the coalesced queue receipt before the successor run can start > - That check required comment ids in the receipt, but a card answer carries its interaction and no comment ids > - So every claim of the successor failed with 403, and the direction never reached a run > - This pull request binds a card-answer receipt to the answer that the successor run delivers, and to the user who gave it > - The benefit is that Interrupt works for queued card answers, and the operator identity stays verified ## Linked Issues or Issue Description Refs #14043, #14510 **What happened** A person accepted a `request_confirmation` card on a task while its run was active, and then clicked Interrupt. The active run stopped ("Interrupted to send queued messages"). The successor run was then cancelled with "Cancelled because the queued run cannot be claimed: Queued-message interrupt authority is unavailable". The coalesced receipt had the interaction id and no `queuedCommentIds`. The successor run's context had the same `interactionId` and the receipt id in `queuedCommentSourceWakeIds`. The chat eval `steering.interrupt-with-direction` recorded this. **Expected behavior** The successor run starts, receives the accepted direction once, and runs with the clicking user as its responsible user. A receipt that the run does not deliver, or an answer that another user gave, still fails closed. **Steps to reproduce** 1. Start a run on a task. 2. While the run is active, accept a confirmation card on that task. The answer goes into the deferred queue. 3. Click Interrupt for the queued messages. 4. Before this change, the successor run is cancelled with `queued_run_claim_rejected`. ## What Changed - `server/src/services/run-identity.ts`: the shared receipt binding (consumed `coalesced` receipt for the same company, agent and run; same task; `user` request from the marker's actor) runs first, as before. Then: - A receipt with comment ids keeps the existing rule: all ids must be delivered to the run. - A receipt without comment ids must be a queued card answer (`queuedInteractionId`). The run context must deliver the same `interactionId`. The interaction must be on the same company and task, with the receipt's status, and `resolvedByUserId` must be the clicking user. Then the run gets the operator identity (`queued_comment_interrupt`). Otherwise the claim still fails with 403. - `server/src/__tests__/heartbeat-queued-run-claim-isolation.test.ts`: a successor of a card-answer interrupt is claimed, runs, and has the clicking user as responsible user. Two negative cases: the run delivers a different answer, and another user resolved the answer. Both stay rejected. Relation to #14510: that open PR fixes the same claim failure (#14043) by returning `null` for card-answer receipts, so the run starts without the operator identity. Its Risks section notes that keeping the operator identity needs a check that the actor resolved the interaction. This PR adds that check instead, so the successor keeps the identity that enqueue time records. If maintainers prefer #14510's approach, this PR's regression test still applies to it. ## Verification - `cd server && npx vitest run src/__tests__/heartbeat-queued-run-claim-isolation.test.ts`: the new positive test fails before the change (successor cancelled) and passes after it. The negative tests and the existing tests pass. - Suites that cover interrupt identity, each run separately: `run-identity` (37), `heartbeat-responsible-user-invariant` (18), `issue-queued-comments-routes` (129), `deferred-execution-receipt-work` (50), `task-interruption-query` (31), `explicit-native-continuation` (436), `run-cancellation` (8), `workspace-recovery-admission` (147), `wake-queue/adapters/postgres` (113): all pass. In `heartbeat-process-recovery`, the interrupt-named tests pass except `dispatches interrupted CLI input and cleans up early fixture exits`, which fails before any interrupt (the CLI child does not start: `controller_lease_lost`). It fails the same way on unchanged master on this loaded host (3 of 3 attempts failed at least one case), so it is not caused by this change. CI covers it. - `server` `tsc --noEmit`: no errors in the changed files. - Live chat eval `chat-features.runner-opencode.local.steering.interrupt-with-direction` at this branch (run 42938970466): passed. The old run stopped with "Interrupted to send queued messages". Exactly one successor run started, delivered the accepted card answer, ran with the clicking user as responsible user, and saved the approved single page and its delivery document. Before this change, the same case failed with the successor cancelled as `queued_run_claim_rejected`. ## Risks - Low risk. The change adds a verified path only for receipts that previously always failed. Comment receipts keep their checks. - If a different user answered the card than the user who clicked Interrupt, the claim still fails. The person who answered can click Interrupt. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude
|
ce3881f1b3 |
fix(skills): release a skill cache lock whose holder's deployment was replaced (#20688)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Run preparation publishes skill revisions into a shared runtime skill cache. A file lock serializes each publication. A lock whose holder provably exited is released at once. > - That check used `controllerProcessExited`, a PID proof. It fails closed whenever the recorded process space (host, kernel boot, PID namespace) differs from this server's. > - Every Railway deploy starts the server in a new process space. A lock left by a server that a deploy stopped mid-publish can never be released. Each later preparation of that skill waits 60 s and fails with "Runtime skill cache publisher is busy … remove it", until an operator deletes the file. > - The rest of recovery already uses one shared prover (`proveOwnerGone`). Across process spaces it accepts a replaced platform deployment on the same exclusive volume, or a stopped or fenced server boot. > - This pull request makes the skill cache lock use that prover. It was the last caller of the separate PID-only proof. > - The benefit is one rule for "the controller from a previous deployment is provably gone", and no manual file removal after a deploy. ## Linked Issues or Issue Description **What happened?** A census of staging failures after deploys showed that `controllerProcessExited` always fails closed after a deploy, because the process-space hash changes. On `master`, the only remaining caller is the runtime skill cache lock check. The restart chaos harness restarts in the same process space, so the PID proof succeeded there and hid the gap. I did not find this failure in the staging run census. It is a latent dead end with the same cause as the census class. **Expected behavior** A lock whose holder ran in a deployment that the platform replaced on the same exclusive volume, or whose server boot stopped or was fenced, is released at once. Any other holder is still waited for. **Steps to reproduce** 1. Run the server on Railway with a volume. 2. Redeploy while a skill revision is being published (the lock file exists). 3. Prepare a run that needs the same skill revision on the new deployment. 4. Observe a 60 s wait and `Runtime skill cache publisher is busy; retry preparation.` on every attempt. **Paperclip version or commit** `master` at `c444da7d05` (the new test fails there). ## What Changed - `server/src/services/runtime-skill-cache.ts`: `runtimeSkillCacheLockHolderExited` builds a local owner stamp from the recorded controller (boot ID, PID, process space, start time, and the Railway deployment and volume when recorded). It returns true only for the `gone` verdict of `proveOwnerGone`. - Same-space PID rules do not change. A dead PID is gone, a live PID with no start time is unprovable, and this server's own boot is alive. - `server/src/__tests__/runtime-skill-cache.test.ts`: a new test for a holder in another process space whose deployment the platform replaced. - The proof is bounded by the lock wait (5 s, never past the deadline). A timed-out or failed proof keeps the lock held. A proof still running is joined by later waits on the same holder, so a stalled read adds at most one database query per holder. - `server/src/services/controller-process-identity.ts`: removes `controllerProcessExited`, which has no caller left; its test assertions now check `sameProcessSpace`. ## Verification - `npx vitest run src/__tests__/runtime-skill-cache.test.ts src/services/controller-process-space-identity.test.ts` (from `server/`): 31 passed. - New: "releases a lock whose holder ran in a deployment the platform replaced". It fails on `master` (`expected false to be true`) and passes with this change. - The same test keeps the lock held for: the same deployment ID, another volume, a non-exclusive volume, another service, and no recorded deployment. - The existing test "never treats age, another host, or a legacy holder as proof that a lock holder exited" still passes. Without a deployment or controller lease proof, another host stays held. - `npx tsc --noEmit -p server/tsconfig.json` passes. - `pnpm verify` selected 2504 of 3294 test files (the change reaches `owner-proof.ts` importers) and hit my 30 min local limit with no failure seen. Required CI supplies the broad coverage. - Live proof: not done (tests only). ## Risks - Across process spaces, a held lock can now trigger a read of `controller_instances` for the holder's boot. This happens at most once per second, and only while a lock is held. Uncontended preparation does no extra work. - The prover may fence a boot row whose lease lapsed past its grace, as it already does for every other recorded owner. - No persisted shape changes. Older lock files without a `controller` field stay held, as before. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude
|
a46b9ac724 |
fix(native): wait out an ACP usage limit's reported reset, then Retry the task (#20692)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs (Paperclip Runner with an ACP harness, such as Claude)
can stop because the provider account runs out of its usage allowance
> - The ACP harness reports this as a "terminal limit failure".
Paperclip blocked the task behind a board hold that called the session
state ambiguous, and the only way forward was a manual Retry
> - The provider also says when the limit resets, for example "resets
Nov 12, 6pm (UTC)" for a weekly limit. Paperclip read only clock-only
resets ("resets 2am"), so the agent-level wait fell back to one hour.
Each later wake for the agent then hit the limit again and blocked
another task
> - This pull request reads calendar resets, says on the hold when the
limit resets, and requests the task's Retry itself at that time when the
reset is reported and bounded
> - The benefit is that a usage limit no longer needs a person to click
Retry, and the agent's other tasks wait instead of failing one by one
## Linked Issues or Issue Description
**What happened?**
A native ACP run failed with `native_provider_usage_limit` ("ACP agent
reported a terminal limit failure. You've hit your weekly limit · resets
Nov 12, 6pm (UTC)"). The task went to `blocked` with a board-owned hold.
The hold said "Inspect the original provider failure and explicitly
resolve the ambiguous session state". Nothing resumed the task after the
reset. The agent-level `provider_limit_wait` recorded "no reset time was
reported, so Paperclip waits one hour", so every hour another wake
dispatched, hit the limit, and blocked another task. In one 48-hour
window on a staging instance, this blocked 7 tasks for one agent.
**What did you expect?**
When the provider reports a bounded reset, the task waits and continues
after the reset, like the existing `provider_limit_wait` path. When the
reset is unknown, the task fails with a clear message and a working
Retry.
**Steps to reproduce**
1. Run a native ACP agent until its provider reports a weekly or session
limit with a reset time.
2. Look at the task: it is blocked with an "ambiguous session state"
hold.
3. Wait past the reported reset: the task stays blocked until a person
clicks Retry.
Refs #20666 (other recovery work in the same staging census; not changed
here)
## What Changed
- `provider-quota-reset.ts`: parse calendar resets such as "resets Nov
12, 6pm (UTC)", "resets Jul 17th at 4:30 PM (America/Chicago)" and
"Resets on August 2, 9am (UTC)". A past date rolls into next year. An
impossible date (Nov 31) is refused. Clock-only parsing is unchanged.
This also fixes the agent-level `provider_limit_wait` reset and the
legacy `provider_quota` reset for these messages.
- New `native-runtime/provider-limit-resume.ts`:
- `readNativeUsageLimitReset` reads the reset only from the run's own
durable events. The last turn event must be a provider-observed
`turn.failed` with a `rate_limit` or `quota_exhausted` provider failure,
and the reset must be at most eight days out.
- `requestProviderLimitResumeRetry` asks for the task's Retry
(`requestTaskRetry`, `initiatedBy: "provider_limit_reset"`) on behalf of
the user who requested the run, or the run's accountable user for
automatic wakes. It is idempotent per run and records its outcome on the
hold. At most three automatic Retries follow one another without a
successful run.
- `registerProviderLimitResumeWork` is a scoped deadline owner (shared
work scheduler, no polling). Each active usage-limit hold is due at its
reset. Startup rebuilds it from the holds, and the executor notifies it
in the hold's transaction.
- `native-session-executor.ts`: for a terminal usage limit, the hold and
run failure detail say when the limit resets and whether Paperclip
retries then, instead of "ambiguous session state". The evidence records
`providerLimitResetAt`.
- `task-retry.ts`: accept `initiatedBy: "provider_limit_reset"`. Its
activity is recorded as system actor `provider-limit-reset` on behalf of
the user, like `restart_recovery`.
- `doc/execution-semantics.md`: document the wait and its limits.
Previous-release story: holds saved by the previous release carry
forward. At startup the owner reads each active
`native_provider_usage_limit` hold. It rewrites the hold's message with
the reset when the reset is known and still ahead, and it requests the
Retry at the reset. Holds whose reset is unknown, more than a day in the
past, or that have no accountable user keep their Retry and are recorded
as not requested, so they are not read again.
## Verification
- `npx vitest run server/src/services/provider-quota-reset.test.ts` — 4
tests. Three fail on `origin/master` code (calendar resets return
`null`), all pass with this change.
- `npx vitest run
server/src/services/native-runtime/native-provider-capacity.integration.test.ts`
— 5 tests on embedded Postgres:
- the existing usage-terminal test (both replay modes) still blocks with
a Retry, and no automatic Retry is requested without a reported reset;
- "waits for a reported usage-limit reset and then requests the task's
Retry once": the hold says when the limit resets, an old-release hold is
rewritten, nothing starts before the reset, one Retry receipt is created
for the accountable user at the reset, and a second pass does nothing.
Fails on `origin/master` executor code;
- "starts the Retry from the hold's own deadline": a real delivery
coordinator rebuilds the deadline from the hold and calls Retry dispatch
once, after the reset;
- "leaves a visible Retry when no user is accountable for the run".
Fails on `origin/master` executor code.
- Focused regression suites pass (642 tests): `task-retry`,
`quarantined-task-retry`, `heartbeat-native-restart-suspension`,
`native-startup-retry.integration`, `provider-failure-classification`,
`heartbeat-provider-limit-wait`, `deferred-execution-receipt-work`,
`chat-run-publications`, `native-runner-restart-recovery.integration`,
`native-remote-retry-admission`.
- `pnpm --filter @paperclipai/server exec tsc --noEmit` passes.
- `pnpm verify` selected a broad run (exit 2, because the docs file is a
global CI input). CI supplies the broad coverage.
## Risks
- The automatic Retry uses the accountable user's authority for
automatic (system) wakes. It is the same Retry the hold already offered
that user, and dispatch still checks Retry authority, stop proof, saved
work and action reconciliation. It is never broader than a click.
- A provider that keeps reporting resets that do not restore capacity
gets at most three automatic Retries in a row; then the Retry stays
visible.
- The deadline owner reads all active usage-limit holds at startup
(limit 500 rows). The number of such holds is small.
- Not changed: the agent-level wait still holds new wakes for at most
one day, as before.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, read-only staging API reads). Extended
reasoning enabled.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude <noreply@anthropic.com>
|