mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-13 15:55:46 +01:00
c45f54ee74a958fb45b56bad6951e4e85fde4e5b
2261
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 DottaandClaude Opus 5.5
|
c45f54ee74 |
fix(native): stop or release unclosable idle warm Runners on environment delete, and say why a close failed (#20957)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs keep a warm Runner in a Daytona sandbox between turns of
a task, so the next turn starts fast
> - Each idle warm Runner has an in-memory owner in the controller.
Deleting an environment closes these owners before it destroys the
sandboxes
> - In every warm chaos wave today, some owners never closed. The server
logged only `{"failed":2}`. Environment delete returned 409 forever, and
all sandboxes of the wave stayed `started` until their 60-minute TTL
> - On staging, that is leaked sandboxes (cost) and owners that block
the work after them
> - This pull request logs and reports the exact close error, stops an
unsettled idle Runner when its environment is deleted, and releases an
owner whose sandbox is already destroyed
> - The benefit is that environment delete finishes or names the exact
run and error, and every warm chaos wave now grades a clean shutdown
## Linked Issues or Issue Description
**What happened?**
After warm Daytona chaos waves (W2, W2Q, W2C cells, `--faults K0`) on
Codex, Claude and OpenCode, the server logged `idle native session
cleanup failed before controller shutdown {"idleSessions":{"failed":2}}`
with no reason. `DELETE
/api/environments/:id?destroyReusableSandboxLeases=true` returned 409
"warm native runner did not shut down cleanly" on every retry. Eight
sandboxes per wave stayed `started`.
**Root cause.**
`closeIdleWarmNativeSessions`
(`server/src/services/native-runtime/native-session-executor.ts`)
dropped the close error (`catch {}`) and counted it. The owners that
failed are the W2Q owners. A W2Q task parks a question and is answered
by a later run on a fresh sandbox. With the reason logged, a wave shows
two causes for the same owner:
1. The owner's close cannot settle (`native_session_close_unrecoverable:
runner did not durably suspend before checkpoint`, the governed-wait
shape from #20844). Only a successor reset could force a stop
(`discardUnsettledSession`). An idle sweep or environment delete had no
forced stop.
2. The owner's sandbox lease was already destroyed by the
pending-cleanup sweep after the successor moved to a fresh sandbox
(#20863). Every close then fails with `Daytona sandbox lease <id> is no
longer active.`, because the stop RPC has no sandbox to reach. The owner
stays registered for ever.
The Runner close memoizes its rejection, so every later attempt fails at
once with the same error. This is not the "controller sequence must be
contiguous" exit (#20946): no close failure in these waves names a
sequence mismatch.
**Expected behavior**
An idle warm owner with no turn in flight never blocks environment
delete for ever. Every failed close names its run and its error.
**Steps to reproduce**
1. Run `tests/restart-chaos/main.ts --env daytona --daytona-warm
--adapters claude --workloads W2,W2Q,W2C --faults K0 --concurrency 8
--tasks 8`.
2. Read `<wave>/sandbox-sweep.json`: `environmentDeleteBlocked` is set
and the sandboxes are `started`.
## What Changed
- `closeIdleWarmNativeSessions` now returns `failures: [{ runId,
issueId, environmentId, providerLeaseId, reason }]`. The reason is the
close error with its code, its settlement record and its cause. It also
logs `idle warm native sessions did not close` with that list. The
idle-timeout close logs its error too.
- `closeWarmNativeSessionsForEnvironment` takes `forceStopUnsettled` and
`db`. The environment delete route sets both. When the graceful close
fails, Core retires the owner as #20844 does: one confirmed Launcher
`stop`, the credential snapshot, `complete`, then instruction
collection. A failed forced stop is added to the reason.
- When retirement still fails and the owner's sandbox lease has a
`destroyed` termination receipt, Core releases the owner. It does not
contact the sandbox. It logs `warm native session's sandbox was already
destroyed; released its owner` with the close error.
- The environment delete 409 now names each run and its reasons, and
`details.nativeRunnerSessions.failures` lists them.
- Idle timeout, controller restart and tenant sleep do not force a stop.
Their conversation can still be resumed, so they only log the reason.
- Chaos harness: new check `warm-shutdown.ts`, "no leaked sandboxes /
clean warm shutdown". Every warm wave (`--daytona-warm` or `--lifecycle
warm`) runs it. It fails the wave on a blocked delete, on a sandbox left
running after cleanup, or on a warm session that did not close at
environment delete or controller shutdown. It writes
`<wave>/warm-shutdown.json` and a section in `SUMMARY.md`.
- Docs: `doc/plugins/PLUGIN_RUNNER_EXECUTION.md` and the chaos README.
## Verification
- RED on master: `native-session-executor.test.ts -t "stops an idle warm
owner whose close cannot settle"` fails: `expected { closed: 0, busy: 0,
failed: 1 } to match object { closed: 0, failed: 1, failures: [...] }`
(a bare count, no reason). `environment-routes.test.ts -t "names the
warm runner session"` fails: the 409 does not name the run. On the first
fix commit, `-t "sandbox lease is gone"` fails: the lease state is never
read, and the owner stays.
- GREEN: those tests pass. The executor suite, environment routes, run
control, recovery, heartbeat process recovery, instance settings, the
idle-sleep, eviction, release-resume, restart-recovery and
warm-checkpoint integration suites pass (11 files, 2471 tests).
`tests/restart-chaos` unit test `warm-shutdown` passes. Server and
harness `tsc --noEmit` pass.
- Daytona warm waves (Claude, W2/W2Q/W2C, K0, 8 tasks). The runner image
`sha256:e04d9ce4…` matches
|
||
|
DottaandClaude Opus 5.5
|
9c56e5465c |
fix(native): keep the agent's own unanswered question open after a turn ends without a report (#20934)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - An agent can ask the person a question and wait for the answer. Native task status comes from the server's status arbiter, not from model prose > - When a native turn ends without `paperclip_finish`, the Runner sends a default report: `done`, objective satisfied, with the agent's final message as the only evidence > - A later comment from the person moves an open question into history (#15392), so the question no longer gates completion > - Together these close a task as `done` while the agent is still waiting for its own answer and has produced nothing. No Retry appears and nothing flags it > - This pull request keeps such a task `in_review` on the agent's own unanswered question when the only completion evidence is the agent's final message > - The benefit is that a "waiting for your answer" turn can no longer become a silent false completion ## Linked Issues or Issue Description Refs #15392 **What happened?** Restart-chaos wave on master |
||
|
DottaandClaude Opus 5.5
|
6cfb1f3d87 |
fix(environments): resume a warm sandbox that its finished predecessor is still releasing (#20953)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A warm Paperclip Runner session keeps its Daytona sandbox between runs. The next run of the same task resumes that sandbox through a `reuse_by_environment` lease. > - Acquisition only resumes a lease that is `released`, `retained`, or held by the same run. A finished run releases its lease only after it saves its instruction and workspace files, which takes several seconds. > - A board wake that arrives in that window does not see the warm lease. It creates a fresh sandbox. The session's durable backup cannot be restored there, because its source lease is still held. The run fails with `native_session_recovery_failed: runner_harness_state_mismatch: backup_without_reusable_lease` and the task needs Retry, which starts a new provider session. > - This pull request makes acquisition wait, for a short bounded time, for a lease that a recently finished predecessor is still releasing, and then resume it. > - The benefit is that the follow-up continues the same warm session in the same sandbox, with no Retry, no restore and no lost provider context. ## Linked Issues or Issue Description **What happened?** Restart-chaos wave `w-20261111T205259-s6301` (Codex, Daytona warm, volume-free server, `--ephemeral-server` from #20930), task t01. The board re-wake run `e7129ec7` started at 20:57:44. Its predecessor `5cf8c306` reported `run.terminal` at 20:57:38 but saved its instruction copy until 20:57:51, so its warm lease was still `active`. At 20:57:53 the new run recorded `startup reuse: environment (fresh)`. It then failed with `runner_harness_state_mismatch: backup_without_reusable_lease` (error code `native_session_interrupted`). Recording the `workspace_finalize` operation then failed with a Postgres deadlock, and the run settled `native_finalization_missing`. Retry run `79aa18ec` worked with a fresh provider session. The K1 restart in that wave was at 20:55:32 and is not the cause: six warm runs after it reused the sandbox normally. **Root cause** `reusableLeaseCanBeResumed` in `server/src/services/environment-runtime.ts` excludes an `active` lease held by another run. That rule is correct for a running predecessor. For a predecessor that has already ended, the `active` state is a short release tail. The new run creates a sandbox, and `retiredSourceAuthorizesNativeBackupRestore` (`native-retired-backup-source.ts`) correctly refuses to restore the backup from a source lease that is not yet `released` or `retained` (`idleWarmSourceRetiresBackup`). The executor's last branch (`native-session-executor.ts`, host preparation, created allocation) then throws `backup_without_reusable_lease`. **How often** Seen in 2 of 361 retained wave result directories on this machine (`s6301` today, `s5201` earlier). It needs a board wake within a few seconds of the predecessor's terminal event, so it is timing dependent. ## What Changed - `server/src/services/environment-runtime.ts`: add `waitForSettlingReusableLease`. Before the plugin and built-in sandbox paths select a reusable lease, acquisition polls (250 ms) for up to 20 s while a matching `reuse_by_environment` lease is `active` under a different run that is `succeeded`, `failed`, `cancelled` or `timed_out` and finished in the last two minutes. It then continues with the normal selection, which resumes the released lease. - A running predecessor causes no wait (unchanged: the new run gets a new sandbox). A predecessor that ended more than two minutes ago and never released causes no wait. - `server/src/__tests__/environment-runtime.test.ts`: two red/green cases (succeeded and failed predecessor release the lease 400 ms after the successor starts acquiring) and one case for an old unreleased lease (no wait, new sandbox, old lease untouched). - `doc/execution-semantics.md`: describe the wait next to the existing warm-reuse rules. ## Verification - Red on master: `pnpm --filter @paperclipai/server exec vitest run src/__tests__/environment-runtime.test.ts -t predecessor` fails both cases with `expected 'warm-2' to be 'warm-1'`: the successor created a second sandbox instead of resuming the first. - Green: the same command passes (3 tests). `environment-runtime.test.ts`, `plugin-warm-tasks.integration.test.ts` and `environment-run-orchestrator.test.ts` pass (288 tests). Server `tsc --noEmit` passes. - Not verified in a chaos wave. The window is a few seconds wide and appeared in 2 of 361 waves, so one wave cannot show the change. ## Risks - A successor can start up to 20 s later when its predecessor's release is slow. Creating a sandbox and restoring a backup takes longer, and failing needs a Retry. The two-minute bound keeps a stuck, never-released lease from delaying every later run. - If a caller held a lock that the predecessor's release needs, the wait ends at 20 s and acquisition behaves as before. - Compared with #20863 (fresh-sandbox fallback) and #20811 (unsaved provider state marker): this change does not alter either. It reduces how often a warm continuation reaches the fresh-sandbox branch. The backup restore proof is unchanged, so an unproven backup still fails closed with a working Retry. - The `workspace_finalize` deadlock in the same run is a separate defect and is not fixed here. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, file edit, test runs). Default context window, extended reasoning on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Devin FoleyandPaperclip
|
6a43890326 |
Fix image attachments in remote native workspaces (#20939)
## Thinking Path > - Paperclip manages AI agents and their work. > - Native runs must receive the files that users attach to their messages. > - Remote wake preparation returned file metadata without readable bytes. > - The historical chat reader also rejected every remote target. > - Command-backed sandboxes and SSH targets already provide a bound command transport. > - This pull request stages verified files through that transport and clears them after the turn. > - Agents can then open attached images with their normal image reader. ## Linked Issues or Issue Description Refs #19740. This change covers SSH and command-backed sandbox execution, including Daytona. Plugin Runner tasks without a command transport still need a separate versioned file-transfer contract. This PR does not close that broader issue. The open attachment-recovery PR #20778 concerns provider-session attachment journals, not input files. **What happened?** A user uploaded an image and sent it to an agent. The remote native run received its filename and size, but no readable file. Wake preparation reported `remote_workspace_staging_unsupported`. The historical read fallback also refused remote execution. **Expected behavior** Copy the exact admitted attachment into the bound remote workspace. Let the provider read the image. Preserve company, task, source-comment, size, and hash checks. Clear temporary file contents when the turn ends. **Steps to reproduce** 1. Select a native agent on a command-backed sandbox or Linux SSH environment. 2. Attach a PNG to its current task or in-app chat message. 3. Ask the agent to inspect it. 4. Check that wake preparation supplies a readable workspace-relative image path. ## What Changed - Add a remote staging owner with bounded commands, stdin transfer, hash verification, opaque paths, and safe image extensions. - Confine Linux filesystem operations through directory descriptors. Refuse symlinks and replacement files during cleanup. - Bind current wake files and authorized historical external-chat reads to the selected lease transport. - Join active writes during cancellation. Retain failed cleanup work until scope close. - Tell providers to open image files with an image-capable reader. - Handle stdin stream errors in the shared SSH transport. Failed uploads return failure instead of raising an unhandled EPIPE or reporting success. - Add database, transport, cancellation, integrity, cleanup, executor-wiring, and Linux filesystem regressions. Document support and limits. ## Verification - Prepared an isolated source checkout with `node scripts/agent-preflight.mjs --skip-rust-debug`. - Focused suites: `pnpm exec vitest run server/src/services/native-runtime/remote-attachment-staging.test.ts server/src/services/native-runtime/native-runner-file-handoff.test.ts server/src/services/native-runtime/chat-attachment-read.test.ts` — 74 passed, 4 Linux-only tests skipped. - Executor checks: `pnpm exec vitest run server/src/services/native-runtime/native-session-executor.test.ts -t 'attachment staging to the remote|stages from the authenticated|verifies frozen staged admission'` — 7 passed. - Server typecheck: `pnpm --filter @paperclipai/server typecheck`, followed by the server layered check after the final changes. - `git diff --check` passed. The local secret, PII, and private-reference diff scan found no matches. - `pnpm verify` returned exit 2 because the documentation change requires broad CI. No full local suite is claimed. - The four Linux filesystem tests were skipped locally because this macOS host has no available Docker daemon. Linux CI executed all 17 remote-staging tests successfully, including those four confinement tests: [server shard 10](https://github.com/paperclipai/paperclip/actions/runs/43102035096/job/130796646266). - SSH regression: the real pipe test reproduced two unhandled EPIPE errors before the fix, plus false success when the peer exited with code zero. The fixed SSH tests and existing argv tests pass (12 tests). The remote-staging transport suite also passes (13 tests; 4 Linux tests require CI). - `pnpm --filter @paperclipai/adapter-utils typecheck` passed after the SSH change. - Final revision `1c5a9288b8d5f267ddeac84638ff6075dbbbf8ec`: all applicable CI jobs passed, including typecheck, build, general and serialized tests, Runner checks, canary dry run, and all eight browser shards. [CI run](https://github.com/paperclipai/paperclip/actions/runs/43102035096). Linux CI also passed all 6 native SSH tests: [server shard 5](https://github.com/paperclipai/paperclip/actions/runs/43102035096/job/130796646349). - Greptile reviewed that final revision at 5/5 with no remaining findings. The review thread is resolved. GitHub reports the PR mergeable with a clean merge state. - No live model turn or deployed release is claimed. ## Risks - Remote staging requires Linux, Node, and a command transport bound to the selected lease. Plugin Runner tasks without that transport retain their unavailable result. - Each file uses bounded independent commands. Slow transports can delay preparation or cleanup. File bytes stay out of command arguments and provider-facing error messages. - An abrupt controller or remote-host failure can bypass cleanup. Ephemeral sandbox disposal remains the backstop. Retained files are never blindly overwritten or deleted. - Previous-release provider sessions and checkpoints retain their format and identity. New turns receive newly staged input files. Existing restart-admission checks still apply. No migration, runner-image change, or account-credential transfer is required. ## Model Used OpenAI `gpt-6.1-sol` through the Codex harness in T3 Code, with xhigh reasoning, repository inspection, code execution, and tests. No subagents were used. The context-window size is not exposed by this runtime. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
757a20d2f1 |
fix(runner): keep remote recovery and launch through a short plugin RPC stall (#20945)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A remote run starts `paperclip-runnerd` in a sandbox. The controller
reads and controls it through the environment plugin
(`environmentExecute`).
> - #20940 made a stalled supervision probe "unknown" while the Runner
stays connected. Other callers still turn the same plugin stall into a
terminal failure.
> - During restart reattachment, the saved-state read, the
identity-marker read and the process check share a 30-second budget. One
plugin RPC that stalls for 40 seconds uses all of it, so a short stall
fails a reattachment to a healthy Runner.
> - At launch, a stalled launch call or identity read fails the run with
`runner_process_termination_unconfirmed` or
`runner_remote_process_identity_unavailable`, although the runner may be
healthy.
> - This pull request treats a plugin RPC timeout as "unknown" and
retries the read-only call within a bounded stall budget. A stalled
launch call is not repeated: the controller reads the launch nonce's
identity marker instead.
> - The benefit is that a short plugin stall during recovery or launch
no longer fails a healthy run. Every positive answer from the sandbox
keeps its meaning, and the bound still fails closed.
## Linked Issues or Issue Description
Refs #20940, #20909
**What happened?**
A plugin `environmentExecute` RPC stall of more than 40 seconds makes a
remote recovery read fail. Plugin RPCs time out at the requested timeout
plus a 30-second overhead, so a 10-second probe can take 40 seconds. The
reattachment read budget is 30 seconds. One stalled read therefore
failed reattachment with `probe_transport_failed` or `probe_timeout`. At
launch, a stalled launch call rejected the run with exit unconfirmed. A
stalled identity read (20-second wait, 2-second reads that can take 32
seconds) failed the run with
`runner_remote_process_identity_unavailable`.
**Expected behavior**
A plugin RPC timeout is a stall of the plugin, not an answer from the
sandbox. Read-only recovery reads retry within a bound. A launch call
that stalls is not repeated. The launch nonce's marker decides whether
the one runner was launched. At the bound, the run fails closed and
Retry works.
**Steps to reproduce**
1. Start a remote (Daytona) run.
2. Restart the controller so that the run reattaches, or start a new
run.
3. While the controller reads the sandbox, stop the plugin worker for 45
seconds.
4. The reattachment or launch fails.
**Paperclip version or commit**
master at
|
||
|
DottaandClaude Opus 5.5
|
612cc7e3e6 |
fix(heartbeat): renew every legacy controller lease in one shared statement (#20924)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A legacy run holds a durable controller lease while it prepares. The server renews the lease every 10 seconds, and it stops the run if no renewal succeeds within the 60 second lease. > - Each run renewed in its own transaction on a private pool of two connections. Each renewal used six round trips (begin, statement timeout, transaction id, notify, update, commit) and woke live-run recovery for that run. > - When many runs prepare at the same time on a loaded database, the renewals queue behind each other. The queue grows with the number of runs, and every run loses its lease at about the same time. > - A healthy server must not fail its own runs because it has many of them. > - This pull request renews all leases of one server process through one shared queue. Requests that arrive while a renewal is in flight share the next statement. > - The benefit is that the renewal cost no longer grows with the number of live runs, so a burst cannot starve the lease. ## Linked Issues or Issue Description **What happened?** On a staging server that did not restart, 22 runs failed with `controller_lease_lost` in about 13 minutes. Every run was a legacy run in the `preparing` stage with `providerWorkStarted: false`. Every loss had cause `renewal_deadline`. The renewal had been pending 28 to 48 seconds (`renewalPendingMs`). The deadline callback ran on time (`deadlineLateMs` 3 to 861 ms), so the event loop was not blocked. The successful renewals before the loss had each taken 20 to 30 seconds, because the deadline fired only 25 to 48 seconds after the last renewal (`sinceRenewalMs`). About 30 runs were live in the process. The first wave started when a user retried about 14 blocked tasks at once. The automatic retries of the first wave then failed the same way, which made the second wave. 12 of 16 affected tasks became blocked. **Steps to reproduce** 1. Start about 30 legacy runs in one server process. 2. Make each renewal transaction slow (for example, a loaded database). 3. Wait one lease period. **Expected behavior** The leases stay owned. The renewal cost does not grow with the number of runs. **Actual behavior** All runs fail with `controller_lease_lost` (`renewal_deadline`). ## What Changed - `server/src/services/legacy-controller-lease.ts`: All controllers of one process renew through one queue per database. One flush is in flight at a time. Requests that arrive during a flush share the next flush, so a renewal waits behind at most one other statement. A flush is one transaction on the private lease pool: `statement_timeout` and one statement that renews all requested rows. - The statement locks the requested rows with `for update skip locked`. A row that another transaction holds is reported busy, and only that run retries after 250 ms. It does not stall the other runs in the batch. A row that is not this boot's live legacy lease is reported rejected, as before. - Renewal no longer emits the `active-execution-recovery` notification. A renewal only moves the deadline later. Recovery already has a deadline at the earlier expiry and rereads the moved deadline then. This removes one recovery pass, `pg_current_xact_id` and `pg_notify` per renewal. - The lease contract does not change: 60 second lease, 10 second renewal, boot-id fence, expired leases cannot renew, the local deadline is measured from the attempt start, and the loss causes and timings are the same. - `doc/run-log-events.md`: Documents the shared renewal queue. Upgrade story: no persisted shape changes. The row columns, lease length and boot-id fence are the same. A lease held by the previous release has a different boot id, so the new batch rejects it and leaves it untouched. A test covers a previous-release lease in the same batch. ## Verification - Red on `origin/master`, green with this change: - `keeps thirty preparing controllers owned when every renewal takes four seconds on a two-connection pool`: before, all 30 controllers lost the lease with `renewal_deadline` (the staging shape). After, none lose it. - `renews fifteen concurrent controllers in one statement` (real Postgres): before, 15 update statements. After, 1. - `renews through one private statement without waking live-run recovery`: before, the renewal woke `active-execution-recovery`. After, it does not. - `live-recovery-work.test.ts` `renews without waking recovery and re-reads the moved controller deadline at the old one`: before, 2 recovery passes per renewal. After, none, and recovery still moves to the new deadline at the old one. - Safety cases: a row lock on one run does not delay another run; a previous-release lease in the same batch stays untouched; a terminated lease connection is retried; existing deadline, retry, rejection and revocation tests pass. - `npx vitest run src/services/legacy-controller-lease.test.ts src/__tests__/live-recovery-work.test.ts`: 88 passed. - `pnpm verify`: see the CI checks on this head. ## Risks - Medium-low. All renewals of a process now share one statement. A failed flush fails every request in it; each watcher retries independently, as it did before. - A busy row is retried every 250 ms until the lease deadline. Before, the renewal waited on the row lock for up to the statement timeout. The loss behavior is the same. - Recovery wakes at the old lease expiry instead of after each renewal. It finds the live lease and moves to the new expiry. - Not in scope: the downstream workspace recovery that blocked tasks after a lost lease (`legacy_execution_requires_reconciliation`, `source_unavailable`). > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (claude-opus-5-5), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
228b0169be |
fix(dispatch): let the answer to the agent's own question continue the task it blocked on (#20931)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - An agent can ask the person a question (`request_human_input`) and wait for the answer > - A later comment from the person moves an open question into history (#15392). The agent still needs the answer, so it reports a blocker and the task becomes Blocked > - The person then answers. Dispatch cancels the answer's continuation with `issue_not_in_progress`, because only a dependency-blocked task may take an answer while Blocked > - The task rests Blocked with a failed run and a Retry. The answer is lost until a person clicks Retry > - This pull request lets the answer continue the task when the same agent blocked it to wait for that answer, proven from durable records > - The benefit is that the person's answer always reaches the agent that asked, without a Retry ## Linked Issues or Issue Description Refs #15392, Refs #20872 **What happened?** Restart-chaos waves with no fault injected fail every W2Q task on every provider (Claude, OpenCode, Codex) with `R3:retry_needed`. For Claude task t03 on master |
||
|
DottaandClaude Fable 5.2
|
b01959cb15 |
fix(recovery): release a handed-off run's saved-files hold for the person it handed the task to (#20941)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - An agent can hand its own task to a person for review. With #20884, a native run that does this keeps running and finishes its turn. > - A restart can land between that hand-off and the end of the same turn. Restart recovery then settles the run as failed and keeps its unsaved sandbox files, so no work is lost. > - Automatic workspace recovery saves those files. But the workspace-repair hold then stayed on the person's task, because the hold only released for a newer agent owner. > - This pull request releases that hold for the person who received the task through the stopped run's own recorded hand-off, once the files are saved. > - The benefit is that the person's task rests in review with no stale "Saved work recovered" hold, and the agent's files are still saved first. ## Linked Issues or Issue Description Follow-up to #20884 (merged). It was reviewed there as a known limit: "Restart blocks the person's task". It uses the `resultJson.humanSelfHandoff` record that #20884 added. Related: #19184, which released the same hold for a newer agent owner. **What happened?** A native run handed its own task to a person, and a restart landed before the run finished its turn. Restart recovery settled the run and retained its files. After automatic recovery saved them, the task still had a workspace-repair hold ("Saved work recovered. Send a message to continue."). The person had nothing to do with that run's work, and nothing released the hold. **Expected behavior** The hold protects the workspace until the files are saved. After that, the task rests with the person and has no execution blocker. **Steps to reproduce** 1. A native run records a hand-off to a person, and the task is now `in_review` with that person. 2. A restart settles the run as failed, with its prepared remote workspace retained. 3. Automatic workspace recovery completes. 4. Before this fix, `getExecutionBlocker` still returns the run's workspace-repair hold. ## What Changed - `server/src/services/execution-blocker.ts`: `restoreHoldReleasedForCurrentOwnerPredicate` now also releases a recovered file-preservation hold in one more case. The task must be assigned to a person, and the stopped run's `resultJson.humanSelfHandoff` must name that task and that person. The existing proof is still required: the completed-recovery receipt bound to the action, and a copyback rather than a snapshot-only save. Until the files are saved, the hold stays. - `server/src/services/workspace-recovery-admission.test.ts`: two embedded-Postgres tests. - The hold blocks the person's task before recovery and releases after it. - A hand-off recorded for a different person keeps the hold. - `doc/execution-semantics.md`: one sentence on the restart case. ## Verification - Red: with the `execution-blocker.ts` change removed, `releases a handed-off run's recovered files hold for the person it handed the task to` fails. `getExecutionBlocker` still returns the hold after `recoverRetainedWorkspace`. - Green: `cd server && npx vitest run src/services/workspace-recovery-admission.test.ts src/__tests__/interrupted-preparation-hold-settle.test.ts src/services/explicit-native-continuation.test.ts src/services/native-runtime/native-human-self-handoff.test.ts`. All 617 tests pass. The server typecheck passes. - Chaos: the H2 workload from #20884 covers the hand-off with a K1 restart after it. Locally the runs had no unsaved files, so no hold appeared. This PR's case needs a remote sandbox with unsaved files. **No Daytona wave was run.** ## Risks - The release applies only when the stopped run's own record names the current person and task, and only after the completed-recovery receipt. Other holds, snapshot-only saves and unsafe archives are unchanged. - No schema change. A run without the record behaves as before. - This branch is rebased onto master after #20884 merged. The only conflict was an adjacent import line. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude (Anthropic), model ID `claude-opus-5-5`, through Claude Code with tool use and code execution. The session does not expose the context window or reasoning mode. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
48d23d1530 |
fix(runner): keep a connected remote Runner through a stalled plugin supervision probe (#20940)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A remote run starts `paperclip-runnerd` in a sandbox. The controller
monitors the process with a health probe that goes through the
environment plugin (`environmentExecute`).
> - A stall of the Daytona plugin worker makes that probe RPC time out
after 40 seconds. The monitor treated the timeout as "process
supervision failed".
> - The transport then failed closed with
`runner_process_termination_unconfirmed`. The run ended
`notification_transport_failed`, `recoverable: false`, and the close
path suspended a healthy Runner. The task rested blocked with Retry.
> - Nothing was wrong with the sandbox, the Runner or the provider. The
Runner held its own authenticated connection to the controller during
the full stall.
> - This pull request makes a failed or timed-out probe "unknown". While
the Runner stays connected on its own channel, the monitor keeps the
same runner and turn and probes again, for a bounded time.
> - The benefit is that a short plugin stall no longer kills a healthy
run. The fail-closed cases stay the same.
## Linked Issues or Issue Description
Refs #20909 (chaos cell CR3, which reproduces this failure)
**What happened?**
A 45-second stall of the Daytona plugin worker killed a healthy run.
Chaos cell CR3 reproduced it 2 of 2 times (OpenCode and Claude,
Daytona). A staging run had the same failure with no deploy: a plugin
`environmentExecute` RPC stalled for more than 40 seconds. The run
failed with "Notification transport failed closed:
runner_process_termination_unconfirmed: runnerd process supervision
failed: RPC call \"environmentExecute\" timed out after 40000ms". The
close path then sent `runner.suspend`, and the healthy Runner suspended
and exited. The task rested blocked with Retry.
**Expected behavior**
A probe that times out or fails in transport is not proof that the
process stopped. While the Runner stays connected on its own channel,
the run continues. If supervision cannot come back within a bound, the
run rests with a working Retry. A probe that positively reports the
process gone still starts the existing recovery.
**Steps to reproduce**
1. Start a remote (Daytona) run with a T1 workload.
2. Send SIGSTOP to the server's Daytona plugin worker for 45 seconds,
then SIGCONT.
3. The run fails `notification_transport_failed` about 40 seconds into
the stall.
**Paperclip version or commit**
master at
|
||
|
DottaandClaude Opus 5.5
|
edd00526e7 |
fix(native): park a response_wake with no admitted response instead of self-waking (#20935)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A native agent run ends with a structured result. The status arbiter
turns that result into an issue status and next actions.
> - A result can report `yielded` with `continuation.kind:
"response_wake"`. This means "wake me when somebody responds".
> - Only a direct Board comment gave this kind a passive wait. Every
other `response_wake` became an immediate `issue_status_changed` wake to
the same agent.
> - The agent woke, saw nothing new, reported `response_wake` again, and
woke again. Nothing bounded this loop. It spent provider budget and its
back-to-back runs caused other failures on warm sandboxes.
> - This pull request parks such a result as a visible wait, and puts
declared continuation wakes under the rewake throttle as a backstop.
> - The benefit is that an agent that waits for input stops and costs
nothing until a real response arrives.
## Linked Issues or Issue Description
**What happened?**
Restart-chaos waves on master `683fd1e9a5` (no faults injected) showed
one task run 30 to 66 times, about 12 s apart. Example: wave
`w-20261111T190352-s5101`, task t02 (W2C), 65 runs in 13 minutes. Each
run reported `yielded` with `continuation.kind: "response_wake"` and the
summary "No changes made. This status-change wake does not trigger phase
2". Each run was woken by `issue_status_changed`, and each finalization
committed `live_continuation_registered` with an `enqueue_continuation`
effect of kind `response_wake`.
Root cause: `arbitrateNativeStatus()`
(`server/src/services/native-runtime/status-arbiter.ts`) only treats
`response_wake` as passive when `boardResponseWaitAuthorized` (from
`readNativeBoardResponseWaitSource`), external chat, or plan wait is
true. Otherwise it falls to the generic yielded branch, and
`status-decision-committer.ts` (`enqueue_continuation`) enqueues an
`issue_status_changed` wake to the same agent. `issue_status_changed` is
not in `THROTTLED_ISSUE_REWAKE_REASONS`, and the native committer writes
an `issue.updated` activity row with the run's id for every decision, so
the rewake throttle saw every native run as progress.
**Expected behavior**
A `response_wake` with no admitted response source parks the task. A
comment, an interaction answer or a status change by somebody else wakes
the agent. No path can loop without a bound.
**Steps to reproduce**
Run `tests/restart-chaos/main.ts --env local --adapters codex
--workloads W2,W2Q,W2C --concurrency 10 --tasks 10 --seed 5101
--lifecycle warm --triggers time`. W2/W2C tasks loop after phase 1.
## What Changed
- `status-arbiter.ts`: a `yielded` / `response_wake` result with no
admitted response source and no blocking remaining work now commits
`response_wake_waiting_for_input`, keeps the task `in_progress`, and has
no effects. Policy version is `phase6-v14`. Board, external chat, plan,
governance, dependency, pause and unfinished-work branches are unchanged
and still come first.
- `native-runtime/native-response-wake-wait.ts` (new) and
`recovery/service.ts`: recovery treats that applied decision as a
durable wait while it is the issue's latest decision, the issue is
`in_progress` with the same agent assignee, and no user comment arrived
after the run started. This applies to the stranded-work sweep and to
the paused-agent passive-wait check.
- `execution-projection.ts`: a committed run with this reason shows the
phase `waiting_for_answer`, label "Waiting for a reply", and a next
action that tells the user to comment or change the status. This shows
only while that decision is still the issue's latest decision and the
issue is `in_progress`; an older run shows "Completed".
- Backstop in `issue-rewake-throttle.ts` and `heartbeat/queue.ts`: an
`issue_status_changed` wake that carries a declared `continuationKind`
(the `enqueue_continuation` effect) is now a throttle candidate. The
native status committer's own `issue.updated` / status-decision activity
no longer counts as run progress or as new input. After two consecutive
successful runs with no issue-visible progress, such a wake waits for
the existing escalating cooldown (2 min, doubling, max 30 min). The
native status dispatcher (`dispatchNativeStatusWakeBatch`) does not drop
a throttled intent: it keeps the intent `queued`, reports the cooldown
deadline to the deferred-receipt worker, and dispatches it again when
the cooldown ends. Status wakes without a declared continuation (blocker
owner, notify owner, retry) are unchanged.
- Restart chaos: new enforced universal check U5 (`universal.ts`). A
task with more than 10 runs fails as `U5:run_budget_exceeded`
(`RESTART_CHAOS_U5_MAX_RUNS`). The README records the calibration.
- `doc/architecture/native-status-arbitration.md`: the decision table
and the wait rules.
## Verification
- RED on master (same reason as the waves):
`heartbeat-process-recovery.test.ts` "parks a native response_wake that
no response admitted…" fails with decision `reasonCode:
"live_continuation_registered"` and effect `{ kind:
"enqueue_continuation", continuationKind: "response_wake" }`.
`heartbeat-issue-rewake-throttle.test.ts` "throttles a
same_agent/response_wake/retry continuation…" fails 3/3 because the wake
is admitted.
- GREEN: the integration test finalizes the run, dispatches native
status wakes and runs the stranded-work sweep three times. It asserts
one run total, one wake request, no recovery action, the task
`in_progress`, and the projection "Waiting for a reply".
- `heartbeat-process-recovery.test.ts` "delays a no-progress native
continuation until the rewake cooldown…" finalizes a real `same_agent`
continuation after a no-progress run, dispatches twice (one throttle
receipt, intent stays `queued`, deadline equals `nextAllowedAt`), then
dispatches after the cooldown and gets the run. It fails when the
dispatcher drops the throttled intent.
- Focused suites pass: `heartbeat-process-recovery`,
`heartbeat-issue-rewake-throttle`, `status-arbiter`,
`native-status-arbiter-corpus`, `runtime-mode`, `execution-projection`,
`chat-run-milestone-work`, `heartbeat-dependency-scheduling`,
`heartbeat-run-summary` (904 tests), and all restart-chaos unit tests
(584 tests). `pnpm --filter @paperclipai/server typecheck` passes.
- Universal audit over the 17 saved waves of 2026-11-11: U5 flags 12
tasks, all looping tasks (12 to 66 runs), and 0 tasks on all-pass waves.
- Chaos rerun of local warm seed 5101 (W2, W2Q, W2C): result is added in
a comment on this PR.
## Risks
- Behavioral shift: a model that reports `response_wake` to mean
"continue now" no longer gets an immediate run. The task waits visibly
for a reply. The runner prompt already describes `response_wake` as a
real response wait, and `same_agent` stays available for server-owned
continuations.
- The throttle now applies to native runs. A chain of declared
continuations with no issue-visible output (no comment, document, work
product or other activity) waits for the cooldown after the second such
run. `same_agent` continuations are server-owned (repository
preparation), so this affects only repeated no-output chains.
- Old saved state: decisions committed under `phase6-v13` keep their
recorded effects. A run that finalizes after deploy uses the new policy.
No migration.
- Budget hard-stop behavior is unchanged.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test runs).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandClaude Fable 5.2
|
b2779054d4 |
fix(issues): let a native run hand its own task to a person without cancelling itself (#20884)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A task has one assignee. An agent often finishes its work by handing its own task to a person for review. > - The task update route stops the outgoing run before it saves a reassignment. > - Since #16082, a legacy run that hands its own task to a person is exempt, so it finishes its turn. Native runs were not exempt. > - On staging, a native run (ACPX Claude, 11.5 minutes of work) sent `PATCH /api/issues/{id}` to hand its task to a person. The route cancelled that run with `issue_reassigned`. The agent saw `api_transport_failure`. The task moved to the person, but it kept a stale execution blocker ("Saved work recovered. Send a message to continue."). > - The native runtime assumes a running run still owns its task. So widening the exemption alone is not enough: the run then fails after its result is accepted, and its finalizer can act on the person's task. > - This pull request applies the exemption to native runs and makes the native terminal fence, restart recovery and finalizer handle a run that gave its task away. > - The benefit is that an agent can hand its task to a person and finish normally, with no blocker on the person's task. ## Linked Issues or Issue Description Refs #16082 (legacy exemption). Related: #20541 (agent-to-agent self handoff) and #18293 (agent key handoff). Those PRs cover other actors and targets. This PR covers native runs that hand their task to a person. **What happened?** A native agent run handed its own task to a person (`assigneeAgentId: null`, `assigneeUserId`, status `in_review`) through its signed run token. The route cancelled the run that sent the request (`issue_reassigned`, "Cancelled before issue reassignment"). The cancelled native run then went through workspace recovery. The person's task kept an execution blocker with `restore_failed`. **Expected behavior** The run that hands its own task to a person keeps running and finishes its turn. The task rests `in_review` with the person. No execution blocker or recovery action stays on the task. The outgoing run cannot change the task after the hand-off. **Steps to reproduce** 1. Run a native agent on its assigned task. 2. From that run, send `PATCH /api/issues/{id}` with `assigneeAgentId: null`, a person's `assigneeUserId`, and status `in_review`. 3. Before this fix, the route cancels the run with `issue_reassigned`. **Why the exemption was legacy-only** #16082 limited the exemption to the run shape of its incident and kept a verified stop for every other case. The new H2 chaos workload showed why native needed more than the route change. With only the route widened: 1. The run's terminal fence `assertCurrentWakeCommentsRead` requires the run to still own the task. It threw `native_current_wake_comments_binding_changed` after the result was accepted. The run then failed with `native_session_lease_lost`, and its finalizer recorded `workspace_finalization_failed` as an agent-owned recovery on the person's task. 2. A restart after the hand-off left the run `running`, because restart recovery refused it with `issue_execution_lock_changed`. 3. The native finalizer only superseded a `done` decision. A `needs_review` report moved the person's task back to `in_progress` and queued a continuation for the outgoing agent. 4. The REST guard against the outgoing run's later mutations (`issue_handed_off`) applied to legacy runs only. ## What Changed - `server/src/routes/issues.ts`: the signed current-run human self-handoff exemption now applies to native runs. All other checks stay: exact run, agent, company and task, `agent_jwt` source, human recipient. For a native run the route first records the hand-off on the run (`resultJson.humanSelfHandoff`, new `native-human-self-handoff.ts`). Board reassignment, agent keys, and handoffs to an agent or to nobody still stop the outgoing run. - `current-wake-comments.ts`: the terminal fence accepts a run whose recorded hand-off matches the person who now owns the task. Without the record (for example a board reassignment), or after a later reassignment, it still refuses. - `native-restart-recovery.ts`: after a restart, a handed-off run is not resumed. It passes the execution-lock check and is then settled by the existing ownership gate (`task_ownership_or_status_changed`) without changing the person's task, instead of staying `running`. - `status-arbiter.ts` and `native-run-finalizer.ts`: `supersedeHandedOffDecision` keeps the person's task status and strips retries, continuations, owner wakes, recoveries, blockers and reviewer bindings, also from decisions that already preserve the status. Two effects stay: a governed request's interaction, and the workspace-finalization repair, so saved work is not lost. Review runs are not affected. - `server/src/run-task-ownership.ts` (renamed from `legacy-run-task-ownership.ts`): `assertHandedOffRunTaskOwnership` rejects later task mutations from the outgoing legacy or native run, in the route and under the update transaction's locks. - `tests/restart-chaos/`: new opt-in workload `H2`. The agent saves a brief, then hands the task to its creator in review as its last step. The oracle requires the task to rest `in_review` with the creator, every run to settle without `issue_reassigned`, readable non-empty run history, and no execution blocker or active recovery action. - Docs: `doc/execution-semantics.md`, task feature map and agent access map, chaos README. ## Verification - Red on `origin/master` (same tests, master source): the route cancels the native run (500, and 409 `runner_goal_reassignment_stop_unconfirmed`); the finalizer moves the person's task to `in_progress` with a continuation for the outgoing agent; a late `done` from the outgoing native run closes the person's task. - Red chaos wave on master's server (`--server-from` a master checkout, `--adapters claude --workloads H2 --faults K1 --matrix`): 0/2 pass. Both runs ended `cancelled` / `issue_reassigned` (`T:handoff_did_not_cancel_run`). - Green focused tests on this branch: - `native-human-self-handoff.test.ts` (embedded Postgres): hand-off through the real issue service; the run stays `running`; `done`, `needs_review` and failed-turn reports leave the task `in_review` with the person, with `getExecutionBlocker` null and no recovery action; the terminal fence accepts only the recorded hand-off; a late change from the outgoing run fails with `issue_handed_off`; pure policy cases. - `issue-agent-mutation-ownership-routes.test.ts`: native and legacy self-handoffs persist without `cancelRun`, and native records the hand-off before the update. Board, agent key, other run, task, agent and company still stop the run. - Native runtime suite (restart recovery, wake comments, arbiter, stale-owner completion), `issues-service`, `low-trust-red-team-routes`: pass. Server and restart-chaos typecheck pass. `node scripts/check-maps.mjs` passes. - Green chaos waves on this branch (local, `--adapters claude --workloads H2`): - `--matrix --faults K1 --seed 4459`: 2/2 pass (K0 baseline and a K1 crossing). - `--tasks 3 --faults K1 --fault-after-seconds 60` and `--fault-after-seconds 100`: 3/3 pass each. In the first, all three restarts landed after the hand-off while the runs were live; the runs settled and the tasks stayed with the person. - The final waves ran on the pre-rebase head `8a2a3b6a74`. The rebase onto master only merged the chaos workload lists (`types.ts`, `config.ts`, `journeys.ts`). No Daytona wave was run. ## Risks - The outgoing native turn keeps running after the person owns the task. Its native task tools already require the run to own the task, so they refuse (`paperclip_runner_tool_binding_not_authorized`). Its REST mutations are rejected with `issue_handed_off`. - A restart after the hand-off settles the run as failed (`native_restart_recovery_blocked`) instead of resuming it. The person's task is not changed and needs no Retry. - The finalizer rule covers any native run whose task belongs to a person at finalization, not only a self-handoff. Before this change such a run could still move the person's task and queue its own continuation. - Saved state: `resultJson.humanSelfHandoff` is a new optional run key. An older server ignores it, so a run handed off on the new server and finished on an old one fails its fence as today. No schema change. - No API shape change. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude (Anthropic), model ID `claude-opus-5-5`, through Claude Code with tool use and code execution. The session does not expose the context window or reasoning mode. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
Nicky LeachandClaude Opus 5.5
|
d7b04a8b6d |
refactor(runner): rename codex-runtime-adapter to acpx-runtime-adapter (#20938)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Paperclip Runner opens agent sessions through one ACPX runtime adapter > - The adapter began as the Codex ACPX adapter, so its file is `codex-runtime-adapter.ts` > - The adapter now opens sessions for every ACPX agent: Claude, Codex, Pi, Cursor, Grok, Kimi, Copilot and Antigravity > - The Codex name makes readers look for Claude and other agent behavior in the wrong place > - This pull request renames the module to `acpx-runtime-adapter.ts` and changes no behavior > - The benefit is a file name that tells readers what the module does ## Linked Issues or Issue Description No issue exists. Refs #20933, which adds Claude restore behavior to this module. **What existing behavior does this improve?** The name of the shared ACPX runtime adapter module in Paperclip Runner. **Subsystem affected** Paperclip Runner, ACPX driver (`packages/paperclip-runner/src/drivers/acpx/`). **Current behavior** The module that opens ACPX sessions for all agents is named `codex-runtime-adapter.ts`. Its test file, which holds Claude, Pi, Grok, Cursor and Copilot tests, is named `codex-runtime-adapter.test.ts`. **Proposed behavior** The module is `acpx-runtime-adapter.ts` and its test file is `acpx-runtime-adapter.test.ts`. All imports, scripts and docs use the new name. **Reason and benefit** The name matches the module's scope. Reviewers and contributors find agent-specific session behavior faster. **Breaking changes** None. No exported name, behavior or contract changes. ## What Changed - Rename `src/drivers/acpx/codex-runtime-adapter.ts` to `src/drivers/acpx/acpx-runtime-adapter.ts`, and its test file in the same way. Git records both as renames with no content change, except the test file's import path. - Update the imports in `acpx-runtime-sidecar.ts`, `codex-acpx-driver.ts` and four test files. - Update the `dist` import in `scripts/grok-native-smoke.mjs` and the test file key in `scripts/vitest-lane-durations.json`. - Update the links and file names in `doc/RUNNER-INTEGRATION-PLAYBOOK.md`, `doc/architecture/runner-cursor-capabilities.md` and `doc/architecture/runner-rich-acp-capabilities.md`. - Do not change the historical profile identity fixtures under `packages/paperclip-runner/test/fixtures/`. They record the sources of earlier releases with the old path, and tests check them by hash. ## Verification - `npx tsc -p tsconfig.json --noEmit` in `packages/paperclip-runner`: no errors in changed files. The only errors are in `src/eval/workflow-harness.ts`, because `@paperclipai/paperclip-eval-kernel` was not built in the local worktree. - `npx vitest run src/drivers/acpx src/cli/acpx-runtime-sidecar.test.ts`: 93 files, 1859 passed, 9 skipped. - `node --test scripts/contracts/contract-fingerprints.test.mjs`: 11 passed. The pack contract digest reads protocol schemas, the Rust crate and the ACPX patch, not TypeScript sources, so the rename does not change it. - `node --check scripts/grok-native-smoke.mjs`, and `scripts/vitest-lane-durations.json` parses. - `git grep codex-runtime-adapter` finds the old name only in the historical fixtures. - Merge check with #20933: a local merge of that branch into this one completed with no conflicts. Git applied #20933's changes to the renamed files. The merged adapter tests passed (133 passed). ## Risks - Low risk. The change renames files and updates references only. - An open PR that edits the old file path must merge or rebase after this PR. Git rename detection handled #20933 with no conflicts. A PR that adds a new import of `./codex-runtime-adapter.js` fails typecheck until it uses the new path. - `openCodexAcpxRuntime` keeps its name. It is an alias of `openQualifiedAcpxRuntime`, and this PR does not change exported names. - `acpx-runtime-adapter.ts` and its test file were already longer than 1,000 lines. This PR does not add lines to them. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file edits, local test runs). Extended reasoning was on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
c1edb9ec02 |
fix(native): keep an unreachable remote Runner alive when a failed run closes (#20923)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs on a remote sandbox (Daytona) start a detached `paperclip-runnerd`. After a recoverable failure, the server resumes the same run and reattaches to that Runner > - On staging, `runner_remote_process_identity_unavailable` was the most frequent failure. The tasks stayed blocked until someone clicked Retry > - In 31 of the 34 failed runs the reattachment found the Runner gone with an unsettled `ready` journal (`runnerSettled: false`). The Runner was not lost by accident. The controller's own close stopped it with `SIGKILL`: close could not prove suspension within its 10 s budget, so it force-stopped the Runner in the middle of a turn > - The server then scheduled a same-run reattachment that could never succeed, because the close had just removed the Runner it needed > - This pull request keeps an unreachable remote Runner alive when a failed run closes. A Stop, a cancelled turn and a `stop_cleanup` recovery still stop it > - The benefit is that the same-run reattachment finds the live Runner. The Runner then settles into a `suspended` journal, which the existing relaunch path resumes. The task continues without Retry, and no provider work is replayed ## Linked Issues or Issue Description Refs #20853, #20907 **What happened?** Remote native runs failed with `runner_remote_process_identity_unavailable` and error code `native_continuation_requires_reconciliation`. The tasks were left blocked. In the failed runs the first attempt failed with a recoverable transport error, for example `provider_transport_failed: PRP command turn.start timed out`, `native_session_interrupted` or a server restart. The run log then showed `runner did not prove durable suspension before checkpoint` and `runner shutdown failed: Error: Durable recovery runner timed out.` The second string comes from `waitForProcess`, which sends `SIGKILL` when its deadline passes. The same-run reattachment ran 20–60 s later and reported `process_verification` / `process_not_running` with `runnerLifecycle: ready` and `runnerSettled: false`. Most of these runs had done provider work before the failure (turns and tool calls), so the automatic Retry correctly refused them (`provider_turn_unsettled`). **What did you expect to happen?** When a run fails for a reason that the server resumes on the same run, the close does not destroy the Runner that the reattachment needs. The reattachment adopts it, and the run continues without Retry and without a repeated tool effect. **Steps to reproduce** 1. Start a remote native run and let a turn start. 2. Make the Runner unreachable (a broken transport or a slow event pump), so that the turn fails and the close cannot prove suspension within its budget. 3. The close sends `SIGKILL` to the Runner. The same-run reattachment then fails with `runner_remote_process_identity_unavailable`, and the task is blocked. **Relation to open PRs** - #20853 covers the 3 runs whose journal was a settled `ready` cut (`runnerSettled: true`). This PR does not change that path. - #20907 makes the Runner's exit reason visible. It does not change recovery. This PR does not overlap with it. ## What Changed - `runnerd-codex-transport.ts`: `close(reason, { retainUnsettledRemoteRunner })` accepts a new option. When the option is set, the Runner is remote (a process launcher or an adopted Runner), the close is not a `stop_cleanup`, the Runner is still alive and suspension is not proven, the close does not stop the Runner. It skips the `waitForProcess` kill, the adopted-runner `SIGKILL` and `forceKill`. It still releases the controller route, writes the unsettled checkpoint and throws `NativeSessionCloseUnrecoverableError`, so the session stays quarantined and is never reused. The settlement diagnostic records `runnerRetained: true`, and a new diagnostic records `remote runner left alive unsettled for same-run reattachment`. - `HarnessNativeSession.close` asks for retention only when the session was not explicitly cancelled. After `cancel()` (Stop), it asks for the old behavior, so Stop still stops the Runner. - `CodexHarnessSession.close` passes the option to the transport. `HarnessSession.close` and `CodexAppServerTransport.close` accept the optional field. Callers that do not set it keep the old behavior. - Docs: `doc/execution-semantics.md` and `doc/run-log-events.md` describe the rule and the `runnerRetained` field. How the run continues: the reattaching controller adopts the live Runner. The Runner then completes the `runner.drain` and `runner.suspend` that the failed close queued, and exits with a settled `suspended` journal. That attempt fails as `native_runner_process_exited`, which the server resumes on the same run. The next reattachment finds the process gone with a settled `suspended` journal and relaunches the Runner over it (the existing path from #19745). If the Runner was only slow, it suspends by itself and the first reattachment relaunches. ## Verification - Red, on `master` with only the tests added: `runnerd-codex-transport-remote-retention.test.ts` fails 2 of 5 (`expected 'gone' to be 'alive'`): the close killed the Runner. `harness-driver-backend.test.ts -t "retain an unsettled"` fails 2 of 2. - Green with the fix: `cd packages/paperclip-runner && npx vitest run src/live/runnerd-codex-transport-remote-retention.test.ts src/backends/harness-driver-backend.test.ts`. - The staging shape: a real `paperclip-runnerd` started through a process launcher, a turn in progress, the Runner frozen with `SIGSTOP` during close. The close throws `NativeSessionCloseUnrecoverableError` with `runnerRetained: true`, the Runner stays alive, and its journal stays `ready`. - A second controller attaches through `adoptExistingRunner` (no duplicate launch). The Runner settles into `lifecycle: suspended` with an empty outbox and no pending terminal delivery, then exits. - An explicit cancellation, a caller that does not ask for retention, and a `stop_cleanup` close still stop the Runner. - Also run: `src/backends`, `src/drivers/codex`, `src/drivers/acpx/codex-acpx-driver.test.ts`, `src/drivers/acpx/codex-runtime-adapter.test.ts`, `src/native-session-runtime.test.ts` (1045 tests pass; 2 `spawnSync ETIMEDOUT` under full-parallel load pass alone), and all of `src/live/` (files that timed out under full-parallel load pass at `--maxWorkers=2`). - `npx tsc --noEmit -p packages/paperclip-runner/tsconfig.json` is clean. - `pnpm verify` selects a broad plan (exit 2) because `doc/execution-semantics.md` is a global CI input. CI supplies that coverage. - Not run: a live Daytona run. The fix is in the controller library that runs on the server, so it needs no new Runner image. ## Risks - Medium. A failed close now leaves an unreachable remote Runner alive instead of killing it. This is the same state a restart detach leaves today, and the same-run reattachment or a Stop cleanup owns it. - If the server does not resume the run on the same Runner, the Runner stays alive until something reattaches or the sandbox is released. Remote Runners have no reconnect deadline (`runnerProcessReconnectGraceMs: null`). Before this change they were killed. The close still quarantines the session, so it is never released for reuse. - Stop is unchanged: an explicit cancellation, `settleCancelledTurn` and `stop_cleanup` close without retention, so they still stop the Runner. - Previous-release story: this change adds and changes no persisted state. State that an older release left (a killed Runner with an unsettled `ready` journal) keeps its current handling: the named refusal `runner_remote_process_identity_unavailable` and a working Retry. Tasks already blocked on staging keep their Retry button. A deploy during a run changes nothing, because the decision is made in the closing process. - No wait is added before `turn.start`. The change only affects the failure path of close. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file edits, test runs, read-only staging API reads). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Fable 5.2
|
e8e548e856 |
feat(db): classify migrations for application under the running release (#20913)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - `packages/db` owns the schema and its migrations. There are 467
migrations, and several ship each day
> - A zero-downtime deploy starts the new release while the previous
release still serves the same database
> - Core does not say whether a migration is safe to apply under the
previous release. A deployer must treat each pending migration as a
reason to pause, or a person must review it
> - On stacks that may not pause, releases wait for hours
> - Core owns the schema, so Core must own this knowledge and publish it
as data
> - This pull request adds a conservative classifier, an in-file
override, the published result, and a guard for new migrations
> - The benefit is that a deployer can apply routine compatible
migrations automatically, and incompatible changes become visible at
review time
## Linked Issues or Issue Description
**Subsystem affected**
Database migrations (`packages/db`), the cloud migrator artifact
(`scripts/cloud-migrator-artifacts.mjs`), and the pull-request policy
job.
**Problem or motivation**
Core has no machine-readable statement that a migration can be applied
while the previous release still reads and writes the database. A
deployer cannot tell an added nullable column from a dropped column. It
must pause for both, or refuse the release.
**Proposed solution**
Classify each migration as `online_compatible`, `requires_staging`, or
`unknown`. Publish the result in the migrator manifest. Fail a pull
request when a migration it adds is `unknown` and has no reviewed
override in the file.
**Alternatives considered**
- A hand-kept list in the deployer. It goes stale and it is not reviewed
with the SQL.
- A sibling JSON file in the artifact. It needs a workflow change, and
the attestation of `manifest.json` would not cover it.
**Roadmap alignment**
This is release infrastructure. It adds no product feature.
## What Changed
- `packages/db/src/migration-compatibility.ts`:
`classifyMigrationSql(sql)` returns `{ compatibility, classified,
reasons, override, overrideError }`. It parses statement by statement
(breakpoints, `;`, comments, string and dollar quoting, guarded `DO`
blocks). A statement that it does not recognize is `unknown`. A
migration takes the worst class of its statements.
- Lock and duration hazards have their own reason codes
(`lock_hazard_*`) and classify as `unknown`, not as compatible with a
warning.
- Override: `-- paperclip:migration-compat online_compatible:
<justification>` (or `requires_staging`) in the leading comment block of
the migration file. The justification is mandatory (10 characters or
more). A malformed, repeated, or misplaced directive is an error and
never improves the verdict.
- A table counts as new after a plain `CREATE TABLE`, or after `CREATE
TABLE IF NOT EXISTS` when no earlier migration created that table.
`classifyMigrationSequence` classifies migrations in apply order with
that knowledge.
- `packages/db/scripts/migration-compatibility.ts` (run with `pnpm exec
tsx` in `packages/db`): classify files, print the distribution, print
JSON, or run the guard (`--check`). No new package script.
- Published data: `manifest.json` of the cloud migrator artifact has a
new additive field `migrationCompatibility`. `version` stays `1`. No
existing field changes. The build reads the SQL from the packed
`db.tgz`, so the data describes what the migrator applies. `validate`
(publisher) and `verify` reject data that is malformed or does not match
the packed migrations. The workflow file does not change.
- Guard for new migrations:
`.github/scripts/check-pr-migration-order.mjs` (the existing "Validate
migration ordering against target branch" policy step) now also
classifies the migrations that the pull request adds. `check:migrations`
runs the same guard against the merge base with `origin/master`, so the
`@paperclipai/db` typecheck and build report it locally. `unknown`
without an override fails. `requires_staging` prints a notice and
passes. A new migration that uses `CONCURRENTLY` also fails when it is
not in `MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS`
(`packages/db/src/client.ts`), because the migrator would run it inside
a transaction.
- `doc/DATABASE.md`: new section "Migration compatibility with the
running release" under "Schema change workflow".
Published shape:
```json
"migrationCompatibility": {
"version": 1,
"migrations": [
{
"idx": 468,
"tag": "0468_busy_wendell_rand",
"sqlSha256": "56a6e272…",
"compatibility": "online_compatible",
"classified": "online_compatible",
"override": false,
"reasons": [
{ "code": "add_column", "compatibility": "online_compatible", "message": "Adds a column that is nullable or has a constant default.", "statements": [1] }
]
}
]
}
```
`sqlSha256` is the hash that the migrator records in the migration
history. `overrideJustification` is present only when `override` is
`true`. Entries are in apply order (journal `idx`, then file name; the
shipped journal repeats `idx` 178).
### Distribution over the existing migrations
No existing migration is rewritten or annotated.
| Class | All 467 | Newest 100 |
| --- | --- | --- |
| `online_compatible` | 228 (48.8%) | 67 |
| `requires_staging` | 114 (24.4%) | 12 |
| `unknown` | 125 (26.8%) | 21 |
Migrations per blocking reason (one migration can have several):
| Migrations | Class | Reason |
| --- | --- | --- |
| 74 | `requires_staging` | `create_unique_index_existing_table` |
| 70 | `unknown` | `lock_hazard_create_index` |
| 66 | `unknown` | `lock_hazard_validated_foreign_key` |
| 56 | `unknown` | `lock_hazard_data_backfill` |
| 54 | `unknown` | `drop_constraint` |
| 41 | `requires_staging` | `add_validated_check` |
| 33 | `unknown` | `drop_index` |
| 16 | `unknown` | `do_block_procedural` |
| 15 | `unknown` | `replace_function` |
| 13 | `unknown` | `create_trigger_existing_table` |
| 12 | `unknown` | `data_change_existing_table` |
| 10 | `unknown` | `drop_object`, `lock_hazard_set_not_null`,
`set_default` (each) |
| 8 | `requires_staging` | `drop_column` |
## Verification
- `pnpm --filter @paperclipai/db exec vitest run
src/migration-compatibility.test.ts`: 113 tests pass. They cover a
positive and a negative case for each rule, each reason code, the
override (missing, short, misplaced, and repeated justification are
rejected), unknown by default, the guard, the shipped migrations, the
`CONCURRENTLY` list, and the command line.
- `node --test scripts/cloud-migrator-artifacts.test.mjs
.github/scripts/tests/check-pr-migration-order.test.mjs`: 21 tests pass.
- Real bundle at the first commit of this branch: `preview-artifacts.mjs
pack`, then `cloud-migrator-artifacts.mjs build`, `validate`, and
`verify-install` pass. The manifest lists 467 migrations and is 312 KB.
This run found the repeated journal `idx`, which the second commit
handles.
- `pnpm --filter @paperclipai/db run check:migrations` passes. With
`--check --base HEAD~150` the guard fails on three `unknown` migrations
and prints notices for two `requires_staging` migrations.
- `pnpm verify` exits 2 because `.github/scripts/**` is a global CI
input. Required CI supplies the broad coverage.
- Not run: the publish job (it needs the master-only AWS role), and no
deployer reads the new field yet.
## Risks
- The manifest grows from about 1 KB to about 312 KB. A consumer that
rejects unknown manifest fields would fail. The verifier in this
repository ignores unknown fields; consumers outside this repository
were not inspected.
- From this merge, an open pull request that adds an `unknown` migration
fails the policy step and the `@paperclipai/db` typecheck until the
author rewrites the migration or adds the override. `drizzle-kit` emits
validated foreign keys and plain indexes for existing tables, so this is
common. The error text and `doc/DATABASE.md` give the fix.
- `ADD CONSTRAINT ... NOT VALID` classifies as `online_compatible`.
PostgreSQL still enforces it on new writes, so the author must make sure
that the previous release does not write rows that violate it.
- The classifier reads SQL only. It cannot see that new code writes
values that the previous release cannot read (for example a new enum
value). The `requires_staging` override exists for that case.
- `master` cannot fail the guard: the guard only judges migrations that
are not on the merge base.
## Model Used
- Claude (Anthropic), model ID `claude-fable-5-2`, in Claude Code with
tool use and code execution.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
|
||
|
DottaandClaude Opus 5.5
|
fc14f1ade2 |
fix(execution): keep resumed native runs alive past a stale finalization deadline (#20920)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The execution control reconciler bounds how long a run may spend
settling after its provider work ends
> - A run deadline from one attempt stayed on the run row after the run
was handed to a same-run resume or a restart claim
> - The reconciler then failed the resumed, live run with
`execution_finalization_deadline_exceeded` and told a person to inspect
the provider
> - This breaks two product rules: recovery must work without a person,
and users must not get operator instructions
> - This pull request makes a live native owner or a scheduled same-run
resume the owner of the run, and it replaces the operator text with
plain wording and Retry
> - The benefit is that runs that recover after a restart keep running,
and a genuine expiry shows plain words and one working Retry
## Linked Issues or Issue Description
No public issue exists. This is the bug description, in the fields of
`.github/ISSUE_TEMPLATE/bug_report.yml`.
**What happened?**
On a hosted deploy, 11 tasks became blocked with
`execution_finalization_deadline_exceeded` in two hours. Three runs had
the same sequence. The native session failed or the server restarted. A
same-run resume or a restart claim then took the run. The new controller
held a live native lease and prepared the run again. The reconciler then
found the earlier attempt's run deadline and failed the run. The
recorded original failure code was empty, so no real failure caused the
block. The run error told a person to inspect the failed run and verify
that its provider stopped.
**Expected behavior**
A run that a live controller owns, or that has a scheduled same-run
resume, continues. A deadline that really expires shows plain wording
and a working Retry. Users do not get operator instructions.
**Steps to reproduce**
1. Save a running native run with an `executionControlDeadlineAt` in the
past.
2. Give its finalization row a live lease (a restart claim), or phase
`retryable_failure` with no lease (a scheduled resume).
3. Run `reconcileAbandonedExecutionControl`.
4. The run fails with `execution_finalization_deadline_exceeded`, and
its error tells a person to inspect the provider.
**Paperclip version or commit**
`master` at
|
||
|
DottaandClaude Opus 5.5
|
34f71f7d57 |
fix(server): defer a run that meets a busy agent directory instead of failing it (#20829)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Each run of an agent with managed agent files gets a private copy of
the agent directory. Remote copies serialize their prepare, checkpoint,
collect, and release operations through one Postgres advisory lock per
agent, taken with `pg_try_advisory_xact_lock`
> - When the lock is busy, the operation fails at once with "Agent
directory operation is already active"
> - On staging, a deploy restarted the controller while three runs of
the same agent were active. When one run resumed, another run of that
agent was checkpointing its directory. The resume failed, used its last
attempt, and the task became blocked with "Retry or send a message"
> - A run must not fail because another run of the same agent is briefly
using the agent directory
> - This pull request keeps the fail-fast lock and, when it is busy
during preparation, puts the same native run back on the existing
same-run retry schedule a few seconds later
> - The benefit is that lock contention delays a run by seconds instead
of ending it as failed
## Linked Issues or Issue Description
**What happened?**
A task's run failed with `native_continuation_requires_reconciliation`
and the error "Agent directory operation is already active". The task
was blocked and asked the user to retry. The same error ended at least
two other runs of the same agent within one day.
**Steps to reproduce**
1. Use an agent with managed agent files in a remote sandbox.
2. Let one run of the agent checkpoint or collect its agent directory.
3. At the same time, let another run of the same agent prepare (for
example, a same-run resume after a controller restart).
4. The second run fails immediately with "Agent directory operation is
already active".
**Expected behavior**
The second run waits until the first operation is done, then continues.
It does not end as failed.
## What Changed
- `server/src/services/agent-directory-working-copies.ts`: The busy lock
now throws `AgentDirectoryOperationBusyError`, still a 409 with the same
message. Lock behavior is unchanged from `master`: a fail-fast
`pg_try_advisory_xact_lock` held for the life of the holder's
transaction.
- `server/src/services/heartbeat/agent-directory-busy.ts`:
`prepareWithAgentDirectoryDeferral` wraps only instruction preparation
and turns the busy error into `AgentDirectoryPreparationDeferral`
("Another run of this agent is still using its files"). A busy lock met
while collecting files after a turn is not wrapped and keeps its
existing path.
- **Before native admission** (a brand-new run, such as the first run of
a task): `server/src/services/heartbeat/retries.ts`
`finalizeAgentDirectoryBusyDeferral` cancels the run as an expected
control-plane wait (`providerWorkStarted: false`) and schedules an
`agent_directory_busy` retry 5–15 seconds later. The task keeps its
execution lock on the retry. The lane spends no failure retries
(`execution-recovery-attempt.ts`), its count is the retry run's
`scheduledRetryAttempt`, and it stops after 30 attempts.
`pre-dispatch-execution-evidence.ts` classifies this wait like the
existing workspace and AI-connection waits, so retry scheduling does not
ask for reconciliation.
- **A resumed native run**:
`server/src/services/native-runtime/native-controller-recovery.ts`
`deferNativeRunForBusyAgentDirectory` keeps the same run on the existing
same-run retry schedule without spending an attempt. It uses the same
guards as `deferNativeControllerRecovery`: it locks the task, requires
the task's execution run to be this run, requires the current lease
owner, and refuses after a Stop, a startup cancellation or a pending
native cancellation, so those settle normally. Its count is stored in
the run's `contextSnapshot`, which a resume claim does not clear, and it
stops after 30.
- `server/src/services/heartbeat.ts`: routes a preparation deferral to
the same-run path or, before admission, to the scheduled-retry path.
After the bound it follows the normal failure path with the plain
message.
- No schema or migration. Postgres releases a dead holder's advisory
lock with its session. The staging evidence showed a live holder
(another run's checkpoint), not a stale one.
- `doc/DEVELOPING.md`: Documented both paths and the bound.
Earlier revisions of this PR waited on the lock and added idle-session
expiry, keepalive, ownership fences, fenced remote removal and attempt
slots. This revision removes all of them.
### How the directory-wait retry keeps the original wake
The retry no longer replaces the wake. `scheduleBoundedRetryForRun`
takes `preserveWakeContext` for this wait: the retry keeps the original
`wakeReason`, `retryReason` and the whole wake context (interaction,
comment, continuation and admission fields), and its wake request keeps
the original reason. The wait is recorded separately as
`preparationWait: { kind: "agent_directory_busy", attempt, nextAttemptAt
}`, next to the carried `executionRetryAccounting`. Gates that key on
the wake reason therefore treat the retry exactly as they treated the
original wake.
| # | Site | Treatment for the directory wait | Why |
| --- | --- | --- | --- |
| 1 | `heartbeat/retries.ts` `scheduleBoundedRetryForRun` | Changed:
`preserveWakeContext` keeps the original wake context and reason and
adds `preparationWait` | Root cause: the retry overwrote the wake reason
|
| 2 | `run-dispatch/domain/wake-context.ts`
`isResolvedInteractionContinuationWakeContext` /
`PREPARATION_WAIT_RETRY_REASONS` | Special case removed | The original
wake context is kept, so the continuation is classified from it |
| 3 | `heartbeat/retries.ts` `interactionRunnerWait` | Special case
removed | The interaction fields are kept in the context |
| 4 | `run-dispatch/adapters/postgres.ts` scheduled-retry dependency
facts | Changed: a preparation-wait retry whose original wake was
admitted as a dependency-blocked interaction
(`dependencyBlockedInteraction`, recorded at wake time) is admitted the
same way | Without the wait the original wake would have run; this gate
is only met because of the wait |
| 5 | `run-dispatch/domain/wake-context.ts`
`isNonAssigneeWorkspaceBusyRetry` + `heartbeat/retries.ts`
`finalizeAgentDirectoryBusyDeferral` | Kept:
`preparationWaitDeferredWhileAssignee`, derived only from the recorded
admission authority | The scheduled-retry ownership gate has no
wake-reason path; the workspace and AI waits use the same recorded flag
|
| 6 | `heartbeat/retries.ts` explicit user continuation admission |
Kept: admits the `agent_directory_busy` scheduling reason | The
successor needs its own authorization receipt |
| 7 | `execution-recovery-attempt.ts` uncharged lane and
`agentDirectoryWaits` | Kept | Cap and accounting |
| 8 | `pre-dispatch-execution-evidence.ts` (`isPreDispatchResourceWait`,
`isAgentDirectoryPreparationWait`), used by
`legacy-execution-recovery.ts`, `cancelled-native-startup.ts`,
`explicit-native-continuation.ts` | Kept | Describe the cancelled
pre-provider parent run, not the retry's wake |
| 9 | `explicit-native-continuation.ts`
`undeliveredLegacyUserCommentIds`, `run-dispatch/adapters/postgres.ts`
`unstartedUserTurnOrigin` | Kept | The cancelled parent did not consume
the user's comment |
| 10 | `execution-projection.ts` label and saved accounting,
`task-interruption-presentation.ts`, UI toast and chat suppression |
Kept | Presentation of the wait |
| 11 | `run-dispatch/domain/policy.ts` AI-wait lock requirement;
AI-subscription lock recheck; AI wait detection
(`execution-configuration.ts`, `provider-limit-wait.ts`, `heartbeat.ts`,
`ai-connections.ts`, `ai-connection-runtime.ts`, `local-ai-login.ts`) |
Not applicable | AI-subscription specific |
| 12 | `workspace-busy-wait.ts`, `environment-selection.ts`,
`workspace-preparation.ts`, UI workspace-holder banners | Not applicable
| Workspace holder parking; the directory wait is a timed retry |
| 13 | `cancelled-native-startup.ts` paused workspace-retry repair;
`failureRetriesBefore*Wait` | Not applicable | Historical workspace and
AI records only; every directory wait saves its ledger |
| 14 | `failedRunRetryBudgetSpent`; storage-wait exhaustion escalation;
`status-relevance.ts` `isAdmissionWait` | Not applicable | A directory
wait never fails the run, stops at its own bound, and its run has
started |
## Verification
- Original wake kinds after a directory wait
(`heartbeat-agent-directory-busy.test.ts`, real heartbeat): an
assignment runs; a plain user comment on a dependency-blocked task is
answered with `dependencyBlockedInteraction`; an accepted interaction
continuation keeps its parked-task and reassignment gates; a user
message continuation runs with its comment; an accepted non-assignee
mention runs; and a reassignment during preparation does not let the
previous assignee restart. Each retry keeps the original `wakeReason`
and records `preparationWait`. With `preserveWakeContext` off, five of
these fail. Restart recovery keeps its attempt credit
(`native-runner-restart-recovery.integration.test.ts`).
- Non-assignee wake: an accepted mention for an agent other than the
assignee meets a busy directory; its retry records
`preparationWaitDeferredWhileAssignee: false`, survives promotion, runs
and succeeds once the directory is free. Without the shared clause in
`isNonAssigneeWorkspaceBusyRetry`, promotion cancels it.
- Interaction continuations: `heartbeat-agent-directory-busy.test.ts`
runs an accepted interaction continuation into a busy directory. The
retry keeps `mutation`, `interactionId`, `interactionKind` and
`interactionStatus` and is still classified as a continuation. If the
task is moved to todo, or reassigned, while the retry waits, the retry
does not run and the task stays parked or with its new owner. With the
old explicit storage/Runner lists, both cases fail.
- Red/green, new run:
`server/src/__tests__/heartbeat-agent-directory-busy.test.ts` runs a
real heartbeat for a new assignment whose first instruction preparation
meets a busy lock. Without the preparation wrapper, the run ends failed
(red). With it, the first run is cancelled as a wait, an
`agent_directory_busy` retry is scheduled with the task still locked to
it, the retry runs and succeeds, and no run of the task is failed
(green).
- Red/green, same-run: with the deferral as a no-op, the same-run
deferral tests failed (`expected false to be true`); they now pass.
- `native-controller-recovery.integration.test.ts` covers repeated
claim-and-defer cycles of a resumed run: each cycle keeps the attempt
count, cannot be claimed before the delay, stores the count on the run,
and the fourth deferral past a bound of 3 is refused. It also covers
Stop after the busy error, a foreign lease owner and a moved task (each
refused, nothing changed), a run before admission, and that a raw busy
error (as from collection) is never deferred.
- Focused suites: 232/232 passed (`agent-directory-working-copies`,
`instruction-working-copy`, `heartbeat-agent-directory-busy`,
`native-controller-recovery.integration`, `heartbeat-retry-scheduling`).
`tsc --noEmit` for `server` reports no errors in the changed files.
- Staging evidence (read-only): the failed run had a graceful restart at
07:25Z, then same-run resumes. Its third attempt failed at 07:31:08Z.
Another run of the same agent recorded a warm checkpoint receipt at
07:31:37Z. A second run of the same agent failed the same way at
07:59:46Z.
## Risks
- A deferred run waits 5 to 15 seconds per deferral, up to 30 times. If
the holder is still busy after that, the run fails as before with a
plain message.
- Before admission, deferral cancels the run and creates a retry run,
like the existing workspace-busy wait.
## Model Used
- Claude Opus 5.5 (claude-opus-5-5), Anthropic, with tool use and code
execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandClaude Fable 5.2
|
a56b648602 |
fix(upgrade-readiness): report restartable runs and a handoff verdict that blocks only on lost work (#20910)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A deployer can replace the server while agent runs continue. Before the swap, it reads the upgrade-readiness report of the old server > - The version 1 report is ready only when every active run is adoptable and no wake admission is in flight > - On a busy instance, new runs arrive each minute. Some runs are always in preparation, so the report is almost never ready and the instance does not upgrade > - A run in preparation has done no agent work. Restart recovery starts it again. A swap loses nothing > - This pull request adds a `handoff` object to the report. It separates recoverable, restartable and blocking runs > - The benefit is that a deployer can wait only for work that a swap would lose ## Linked Issues or Issue Description Refs #20852 (reference only; this change does not depend on it and does not use its version 2 shape). **What happened?** A busy instance had 16 running runs and 1 queued run. Fourteen had a live process. Two or three native runs had not launched a process yet. One was a legacy run. `upgradeReadiness.ready` needs `blockingRuns === 0 && pendingWakes === 0` in one probe. That state almost never occurs, so the deployer never swaps the server. **Expected behavior** The report tells the deployer which runs a swap would lose. Runs that only wait for a launch, and wake admissions in flight, do not block a graceful replacement. **Steps to reproduce** 1. Keep a steady flow of new runs on an instance. 2. Read `GET /api/instance/task-drain?upgradeReadiness=1` repeatedly. 3. See `ready: false` each time, because one or more runs are in preparation or a wake admission is in flight. ## What Changed - `summarizeUpgradeReadiness` returns an additive `handoff` object: `version`, `ready`, `recoverableRuns`, `restartableRuns`, `blockingRuns`, `pendingWakes`, `unsavedWakes`. - `handoff.ready` is true when the instance is not draining, the census is complete, and `handoff.blockingRuns` and `handoff.unsavedWakes` are 0. Saved pending wakes do not change it. - The wake queue tracks each public wake admission until its wake request and queued run commit. `unsavedWakes` is the count of admissions before that commit. Without a valid count, every pending wake is unsaved. - The top-level version 1 fields keep their values and meaning. `ready` and `blockingRuns` are unchanged, so a current consumer sees no looser verdict. - The census query marks a run `restartable` only with all of these: `paperclip_runner` adapter; no process identity; no provider session before or after; no checkpoint header; no dispatched runner task; no execution-start event. It must also be a native run in the `observed` phase with preparation fencing and an unsettled finalization record, or a legacy run in the `preparing` stage under a recorded controller. - All other runs without adoption evidence stay blocking. This includes local adapters, legacy runs with a child process, follow-up runs that carry a provider session, and runs the census could not observe. - The OpenAPI description and `doc/testing-native-remote-continuity.md` describe the new fields and their limits. Why these classes are correct on master: - Restartable native runs: restart recovery claims a run with no checkpoint and no provider evidence as `bootstrap_incomplete` and executes the same run again (`native-restart-recovery.ts` `classifyNativeRunnerRecoveryEvidence`; `heartbeat/recovery.ts` executes each claim). A graceful shutdown suspends the run and notifies recovery. Existing tests: "recovers an unclaimed remote startup once without requiring a provider checkpoint" and "recovers an unclaimed remote startup that a graceful shutdown suspended during preparation". - Restartable legacy runs: process-loss recovery retries a `preparing` run with `providerWorkStarted: false`. Existing test: "continues a chain whose preparations were killed twice and then interrupted again, without a Retry". - Pending wakes: the count is wake admissions in flight in the process, not queue rows. A wake commits its wake request and its queued run in one transaction. After the commit, the next server dispatches the queued run at startup, so a saved wake does not block. Before the commit, nothing is saved. Shutdown waits at most 30 seconds and then exits, so an unsaved wake can be lost and blocks. ## Verification - `cd server && npx vitest run src/services/heartbeat/upgrade-readiness.test.ts src/services/heartbeat/upgrade-readiness-post-result.test.ts`: 28 passed. - Red/green: before the implementation, the 7 new tests failed (6 summary tests, 1 PostgreSQL census test). After it, all pass. - Red/green for unsaved wakes: a test holds the agent row lock while a wake admission waits. Without the fix, `handoff.ready` was true with no saved wake. With the fix, it is false with `unsavedWakes: 1`, and the wake saves after the lock releases. - New cases: only recoverable runs; recoverable plus unlaunched runs plus pending wakes (`handoff.ready` true, top-level `ready` false and `blockingRuns` unchanged); a started run without recovery evidence; an unobserved execution; draining; truncated census. - The PostgreSQL test changes one fact at a time (process identity, provider session, phase, fencing, checkpoint, adapter, each execution-start event) and expects the run to become blocking. - `cd server && npx vitest run src/__tests__/openapi-routes.test.ts src/__tests__/instance-settings-routes.test.ts`: 157 passed (187 with the two readiness files). - `cd server && npx tsc --noEmit`: passed. - `pnpm verify` exited 2 (a documentation path is a global CI input). Required CI supplies the broad coverage. - Not run: a live server replacement. The classification relies on the existing recovery tests named above. ## Risks - Low risk for current consumers. The change is additive and the version 1 fields are unchanged. - `handoff.ready` is an observation at one instant. A wake admission that starts after the probe is unsaved until it commits. - Restart recovery applies its own proofs and a bounded attempt budget before it starts a restartable run again. The report is an observation, not a guarantee for each run. - The probe adds one indexed `not exists` on run events per running row, inside the existing two-second statement timeout. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Fable 5.2 (`claude-fable-5-2`), with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
7e8901aa60 |
fix(recovery): let Retry and the person's answer continue a held task with a pending decision (#20872)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run on a remote sandbox can lose its Runner process.
Recovery then saves the sandbox files and puts an execution hold on the
task
> - The hold offers Retry. A person can also continue the task with a
new message
> - Many held tasks also have a pending decision: the question that a
parked turn asked, or an older connection request
> - Continuation admission refuses every continuation while a decision
is pending ("decision_pending"). The answer to that decision is not a
message, so admission defers it behind the hold ("execution_recovery")
> - So Retry waits for the answer and the answer waits for the hold. The
task shows a Retry that does nothing, and nothing continues it
> - This pull request lets a person's Retry pass an open question, and
lets the person's own answer continue the task through the hold
> - The benefit is that a task with a lost Runner and a pending decision
always has a control that works
## Linked Issues or Issue Description
**What happened?**
On staging, the same-run retry of six remote native runs found the
Runner process gone (`process_not_running`, lifecycle `ready`). Each run
failed with `runner_remote_process_identity_unavailable`. Saved-work
recovery then recovered the sandbox files, and the task showed "Saved
work recovered. Retry or send a message to continue." with `canRetry:
true`. Four of these tasks also had a pending decision: the question
that the parked turn asked, or an older pending connection request. For
those tasks, Retry is deferred with `decision_pending`, and an answer to
the decision is deferred with `execution_recovery`. No control continues
the task.
**What did you expect to happen?**
Retry starts one new run. The person's answer to the pending decision
also continues the task.
**Steps to reproduce**
1. A remote native run asks the user a question and parks the turn.
2. The Runner process exits. The same-run retry fails with
`runner_remote_process_identity_unavailable`.
3. Saved-work recovery recovers the files. The task offers Retry.
4. Click Retry: the Retry waits with "A pending approval or question
must be resolved before this message can start."
5. Answer the question: the answer waits with "Waiting for execution
recovery. Your message is saved."
Refs #20838, #20853
## What Changed
- `admitExplicitNativeContinuation`
(`server/src/services/explicit-native-continuation.ts`): a task Retry is
no longer held by a plain pending question (`ask_user_questions`) or
connection request (`connection_intent`). Any interaction with approval
semantics still holds Retry: a plan review (`issue_document` target), a
governed tool action, a secret proposal, a connection authorization, and
other confirmations. A pending approval still holds every continuation.
A new message supersedes a plain question, as on master, and waits for a
decision with approval semantics.
- `admitExplicitNativeContinuation`: the person's own resolution of an
interaction (`mutation: "interaction"`, answered/accepted/rejected,
resolved by the same user) is explicit user input. It continues the held
task like a new message: a fresh session, the hold resolved as
`explicitUserContinuation` with the `interactionId`, and
`issue.execution_recovery_settled` activity. It never replays the failed
run. Governed tool-action confirmations are excluded. The existing
`readQueuedInteractionResponse` reads the answer.
- `heartbeat/queue.ts`: both admission calls pass the wake payload to
the admission.
- `buildExecutionContinuation`
(`server/src/services/execution-continuation.ts`): dispatch rechecks the
user authorization. It accepted an interaction receipt only through a
saved interrupt queue, so the answered successor failed with
`continuation_user_authorization_missing`. It now also accepts the
person's own answer: the interaction named by the durable authorization
was resolved by that user and not by an agent, and the authorization
names this successor run. The check does not depend on the run's current
trigger, so a later answer that replaces the trigger cannot invalidate
it.
- New API-level test
`server/src/services/governed-wait-runner-loss.test.ts` with the two
staging shapes (a parked question from the failed run; an older
connection request on a monitor wake). It checks that Retry starts
exactly one run and leaves the decision pending, that the person's
answer starts exactly one run and settles the hold, and that an answer
resolved by a different user keeps the hold.
- `doc/execution-semantics.md`: describes the rule.
## Verification
- `cd server && pnpm exec vitest run
src/services/governed-wait-runner-loss.test.ts`: 5 passed. The answer
tests now also build the successor's continuation as dispatch does and
check that the agent receives the answer.
- Red for dispatch (first head,
|
||
|
Nicky Leach
|
25f0957c24 |
Select the AI Connection sign-in computer automatically (#20906)
## Thinking Path > - Paperclip manages agents and their AI Connections. > - Codex and Claude subscription sign-in can run on a company agent computer. > - Core stores the credential and uses connection permissions to control access. > - The Connections page asks the user to choose a computer just to host sign-in. > - This change lets Core choose an eligible computer when sign-in starts. ## Linked Issues or Issue Description Refs #20410 and #20417. **What existing behavior does this improve?** OpenAI and Anthropic subscription sign-in from Connections with an agent computer as the sign-in environment. **Current behavior** The user must choose a sign-in computer before the provider flow can start. **Proposed behavior** Start sign-in from the provider form. Core chooses a computer in the company and sign-in environment. The session stays bound to that computer and reports when no eligible computer is available. **Reason and benefit** The saved credential and its permissions are independent of the computer that hosts sign-in, so users can proceed without picking an agent. ## What Changed - Select an eligible computer when a managed Codex or Claude login starts without an explicit target. Prefer ready computers and skip active login leases. - Remove the picker and let the browser resume the existing session by environment and connection intent. Continue to honor explicit agent targets in agent settings. Remove the unused picker fallback from the legacy form. - Update the AI Connections guide, the plugin environment contract, and the OpenAPI descriptions. ## Verification - Passed server and UI typechecks. - Passed 317 tests across the login environment, setup-token route, device-login service, and two UI suites. The Postgres test covers eligibility, inherited environments, explicit targeting, occupied computers, and the no-computer response. - Passed 45 OpenAPI route tests. - Reran the affected UI suites on the final revision. All 230 tests passed. - UI tests cover starting from the provider form and resuming both provider flows with a server-selected target. - [CI](https://github.com/paperclipai/paperclip/actions/runs/43086849302) and [Greptile 5/5](https://github.com/paperclipai/paperclip/pull/20906#issuecomment-6805135968) passed on the final commit. - Local verification used test transports. Live provider sign-in remains to be checked in a Cloud composition that contains this change. ## Risks - Two sign-ins that select a computer before either lease is recorded can contend for the same computer. The existing provider admission guard handles the conflict. A retry can select another computer once the lease is visible. - The computer selection uses existing lifecycle and lease data. Plugin admission still validates the live computer. - No schema migration or plugin protocol change is required. ## Model Used OpenAI Codex (GPT-6), with shell execution and local test tooling. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge |
||
|
Nicky LeachandPaperclip
|
5dfbb04b85 |
Poll asynchronous workspace provider preparation (#20865)
## Thinking Path > - Paperclip starts agent runs through a workspace provider when a project uses remote storage. > - The provider must prepare the workspace before the agent starts. > - One long worker call ties the checkout to its transport. > - Some checkouts take longer than the transport allows. > - This change lets an opted-in provider accept work and report its deadline. > - Core waits for completion through short status calls and stops at the provider deadline. ## Linked Issues or Issue Description **What happened?** A workspace provider can lose its reply when preparation takes longer than a single transport request. This stops run bootstrap before agent execution. **Expected behavior** Core should wait for a provider-confirmed workspace through short requests. The provider's deadline must bound the wait. **Steps to reproduce** Use a workspace provider that takes longer to prepare a checkout than its transport can remain open. Start a run that requires a new checkout. ## What Changed - Add an optional workspace preparation method to the plugin SDK. - Validate pending receipts and keep the original binding checks on completed replies. - Poll only during active preparation. Retry a lost status reply within the advertised deadline. - Preserve the original operation for providers that do not offer asynchronous preparation. - Reject deadline offers that exceed the existing lifecycle lock budget. Honor accepted offers without shortening them. - Checkout verification uses the existing synchronous RPC with a 10-second deadline and a 12-second transport timeout. [Core verification PR #20876](https://github.com/paperclipai/paperclip/pull/20876) ## Verification - All 63 workspace SDK and Core integration tests passed with temporary local PostgreSQL. The tests cover lost status replies, deadline expiry, oversized offers, and synchronous verification with an async-capable worker. - All 58 runner SDK input, evidence, and public type tests passed with the regenerated source pins. The local server typecheck passed. ## Risks - This change activates only when the provider advertises the method. Existing providers keep their current behavior. - Polling uses one-second intervals during active preparation. It stops at completion, failure, or the provider deadline. - No database migration is required. ## Model Used - OpenAI GPT-6 Codex with local code execution and tests. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
8eabb13ebd |
fix(recovery): retry a system-woken run once when it lost its runner before any turn (#20873)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run on a remote sandbox can lose its Runner process before
its same-run retry reattaches. The run then fails with
`runner_remote_process_identity_unavailable`
> - After saved-work recovery, the server requests one Retry by itself,
but only for user wakes. System wakes (monitors, automatic assignments)
rest on the visible Retry
> - On staging, a monitor wake's run lost its Runner before it started
any provider turn. Nothing it did could be repeated, but the task stayed
blocked until a person clicked Retry
> - This pull request lets that one case continue automatically, once,
under the run's responsible user
> - The benefit is that a system-woken task whose run never reached a
provider turn continues without a person
## Linked Issues or Issue Description
**What happened?**
On staging, a monitor wake (`issue_monitor_due`) started a remote native
run. Its `session.snapshot` command timed out before any turn. The
same-run retry found the Runner process gone (`process_not_running`,
lifecycle `ready`) and the run failed with
`runner_remote_process_identity_unavailable`. The run had no turn event
and no tool event. Saved-work recovery recovered the files, but
`requestInterruptedNativeTurnRetry` refused with `not_user_request`, so
the task stayed blocked on a manual Retry.
**What did you expect to happen?**
When the run proves that it did no provider turn, the server requests
the Retry once by itself, as it does for user wakes.
**Steps to reproduce**
1. A system wake starts a remote native run.
2. The run fails before any turn event, and the same-run retry finds the
Runner gone.
3. Saved-work recovery completes. No Retry is requested; the task waits
for a person.
Refs #20872, #20853
## What Changed
The rule that changed, exactly: in `requestInterruptedNativeTurnRetry`
(`server/src/services/heartbeat/restart-interrupted-turn.ts`), a run
whose wake was not requested by a user used to stop with
`not_user_request` in all cases. Now it requests the task Retry only
when all of these are true:
- the run failed with one of the existing reattachment-loss codes
(unchanged gate);
- a positive durable record proves that no turn was ever issued: the
run's last `native.remote_runner_recovery_failed` event carries
`remoteRunnerRecovery` with `stage: process_verification`, `reason:
process_not_running` and `providerTurnIssued: false`;
- the run also recorded no turn event and no tool event;
- the run has a `responsibleUserId`; the Retry runs under that user;
- the run is not itself a `task_retry` run;
- the task is still open and assigned to the same agent (unchanged
gate).
An absence of run events is never enough: a turn can reach the provider
before its events are stored (Greptile, first review).
- New `withRemoteRunnerTurnProof` (`native-session-executor.ts`): when a
same-run reattachment is refused because the sandbox proved the Runner
process gone (exit 45, not an unreachable sandbox), the executor reads
the run's controller journal. The controller records each turn command
there before it sends it. Only when
`runnerdStateProvesNoRemoteProviderTurn` holds (just
`run.prepare`/`session.open` and maintenance commands such as
`session.snapshot`, and only session-open events) does it add
`providerTurnIssued: false` to the diagnostic. This is the same proof
that restart recovery already uses for `bootstrap_incomplete`.
`RemoteRunnerRecoveryDiagnostic` keeps the field only as `false`.
- Which staging shapes this covers: a run whose journal holds no turn
command. Whether c14bab19 (the `session.snapshot` timeout) qualifies
depends on its controller journal, which staging does not expose; if its
journal held a `turn.start`, it stays on the manual Retry floor.
32376b3a (`turn.submitted`, then a `turn.start` timeout) always stays on
the floor.
- Everything else about system wakes is unchanged: without the journal
proof, with any turn or tool event, or without a responsible user, a
system wake still rests on the visible Retry.
- Bounded: the Retry uses the existing idempotency key
`restart-interrupted-turn:<runId>`, so it is requested at most once per
run. Its successor is a user-attributed `task_retry` run, so if that run
also loses its Runner, the existing `automatic_retry_used` rule leaves
it on the visible Retry. No loop is possible.
- Attribution: `requestTaskRetry` records `issue.retry_requested` with
actor `restart-recovery` on behalf of the run's responsible user.
Admission keeps every gate (saved-work recovery, stop proof, approvals,
pause, budget, ownership).
- Tests in
`server/src/services/workspace-recovery-retry-copyback.test.ts`: a
positive case (monitor wake, Runner proven gone, journal proof),
including the second-loss `automatic_retry_used` check. Negative cases:
no run events but no journal proof (the Greptile shape), a settled turn,
a tool event, no responsible user.
- New
`server/src/services/native-runtime/remote-runner-turn-proof.test.ts`:
the proof is recorded only for a journal of session commands; not for a
`turn.start` command, a committed turn event, an unreadable journal, or
an unreachable sandbox (`probe_transport_failed`).
- `doc/execution-semantics.md`: describes the rule.
## Verification
- `cd server && pnpm exec vitest run
src/services/workspace-recovery-retry-copyback.test.ts`: 83 passed.
- Red on master (product hunk removed): the positive case fails with
`expected { requested: false, … } to match object { requested: true }`.
- Red on the first head (
|
||
|
DottaandClaude Opus 5.5
|
566a97b882 |
fix(native): an unretirable predecessor sends the next attempt to a fresh sandbox (#20863)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs on a reused sandbox keep the previous run's warm Runner, and a new run must retire that owner before it starts > - #20844 retires the owner when Paperclip can prove it stopped. Some owners can never be proven stopped: the controller journal drain outlives the close grace, the Runner never confirms suspension, or a parked question leaves a tool input unanswered > - Then the new run failed before any provider work, and the task stayed blocked with no automatic continuation > - The general answer is to leave the old Runner on its sandbox and start the next attempt on a fresh sandbox restored from the saved workspace > - This pull request adds that fallback for every path that retires a predecessor, and settles a parked question's tool input once its Runner is gone > - The benefit is that an unprovable predecessor no longer fails a task that did no work ## Linked Issues or Issue Description **What happened?** An automatic wake started a new run 30 to 60 seconds after a long successful run on the same task, on the same reused sandbox. The new run had to retire the previous run's warm session first. The retirement failed and the run failed before any provider work, with one of: - `Controller journal drain timed out; preserve the session before replacing its writer.` (during instruction preparation, before runtime selection, or in the executor); - `provider_transport_failed: runner did not durably suspend before checkpoint` (during instruction preparation); - `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate; retirement_refused=journal_tool_input_unanswered` (the previous run had parked on a question). The tasks rested `blocked` with a reconciliation hold or with nothing scheduled. One bounded retry reused the same sandbox and ended `execution_finalization_deadline_exceeded`, then its recovery refused with `allocation_ownership_ambiguous`. **Expected behavior** A run that did no provider work does not fail the task because its predecessor cannot be retired. The next attempt drops the reused sandbox and starts on a fresh one restored from the saved workspace. Unsaved work is never dropped: when the workspace or the session backup is not durably saved, the sandbox is kept and today's behavior applies. **Steps to reproduce** 1. Complete a warm native run on a Daytona sandbox. 2. Wake the task again so that the next run must retire the warm owner (configuration change, new session, changed agent files). 3. Make the owner's close fail (journal drain timeout, unconfirmed suspension). 4. The new run fails before provider work and the task is not continued. Refs #20844, #18373. ## What Changed - `native-runtime/predecessor-retirement.ts`: a typed pre-provider error `NativePredecessorRetirementUnprovenError` (`native_predecessor_retirement_unproven`) and the `replaceBeforeReuse` lease marker. - Native executor: every path that retires a predecessor for a new run (instruction-copy claim, instruction-directory reservation, session reset, configuration change) throws the typed error when the close fails. The owner stays registered for its own sandbox, so no run on that sandbox launches over it. A run on a different sandbox forgets that owner instead of replaying its cached rejected close. The executor does not settle this error as a provider failure. - A prior-owner fence that refuses a terminal owner on a resumed sandbox (`prior_owner_terminal_state_indeterminate`) also throws the typed error. - Retirement of a finished remote owner accepts an unanswered tool input when the owner's committed result is a governed-wait park (`response_wake` on a task interaction) and its Runner's sandbox lease ended elsewhere. - Heartbeat (`heartbeat/predecessor-retirement-fallback.ts`): on the typed error with no execution-start evidence and no result, mark this run's reusable lease on that sandbox `replaceBeforeReuse` and log `environment.lease_replacement_requested`. After the run ends, `failRun` hands the prepared workspace on only through the shared unchanged-files proof (`proveRemoteWorkspaceUnchanged`; a provisioning command or any runner command refuses it). With the proof it records `predecessorRetirementFallback`, closes the run's coordinator without a result and re-queues the run in the uncharged `interrupted_preparation` lane (native and legacy runtime modes); a run that answered a task interaction keeps its own wake in that uncharged lane, so the retry still carries the answer. Without the proof the workspace stays dirty, heartbeat runs today's failed-workspace preservation and no retry is scheduled. - Environment runtime: a marked lease is replaced instead of resumed, under the same rule as a failed resume (`reusableSandboxReplacementIsSafe`: run ended, clean workspace with a committed save, verified harness backup). The old lease goes to `pending_cleanup` for the sweep. Otherwise the sandbox is resumed as before. - Chaos: session fault SR1U, the SR1 variant where the retained owner cannot be retired. The server gets a test-only fault (`PAPERCLIP_TEST_WARM_RETIREMENT_FAULT=unconfirmed`, armed only when `NODE_ENV=test`). The oracle allows only the fallback failure and requires the succeeding follow-up on a different sandbox. - Docs: `doc/plugins/PLUGIN_RUNNER_EXECUTION.md`, chaos README. ## Verification - RED on master before this change (with #20844), then GREEN: - `native-session-executor.test.ts -t "predecessor cannot be retired"`: base fails with `Error: Controller journal drain timed out…` instead of the typed error. - `native-session-executor.test.ts -t "unretired predecessor"` (instruction preparation): same base failure. - `native-recovered-owner-followup.test.ts -t "unacknowledged"`: base fails with `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate; retirement_refused=journal_tool_input_unanswered`. - New contract tests: `heartbeat-native-predispatch-retry.test.ts` (claimed native run: lease marked, activity logged, uncharged `interrupted_preparation` retry, no recovery action, task not blocked, retry succeeds) and `environment-runtime-reusable-resume-replacement.test.ts` (a marked lease is replaced without a resume; with unsaved work or no verified backup it resumes as before), `remote-workspace-unchanged-release.test.ts` (an untouched workspace is handed on; a setup-written one stays dirty) and `retries.test.ts` (an answered interaction re-queued in the uncharged lane stays a resolved-interaction continuation with no failure retry spent). - Existing suites pass: the native executor, recovered-owner follow-up, instruction working copy, run completion, run failure cause and resume replacement test files. `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts question-handoff scheduler config` passes. Server and chaos harness `tsc --noEmit` pass. - Not run: a Daytona SR1U wave. It needs a runner image that matches this checkout. ## Risks - A sandbox marked for replacement is resumed when it cannot be replaced safely. That run can meet the same unretired owner again, fail again in the bounded lane, and then rest with Retry, as today. - The replacement session restores the latest verified harness backup. When the old Runner's final checkpoint was never saved, a same-session continuation resumes from that backup. - An unanswered tool input is accepted only for a committed governed-wait park whose Runner's sandbox lease ended on another sandbox. - Previous-release story: the marker and the run record are additive JSON. An older server ignores the marker and resumes the sandbox as before. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, agentic coding with tool use (shell, file edits, test runs) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Fable 5.2
|
ffe05b1b7c |
fix(heartbeat): retry every pre-provider lease acquisition failure and rest it on Retry (#20857)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A task run takes an environment lease (for example a Daytona sandbox) before the agent starts. No provider work happens before that lease exists. > - Only a provider HTTP status or a connection errno counted as a transient lease failure. #20828 (merged while this was open) added the Daytona acquisition deadline. A database error while the lease was recorded, a plugin RPC deadline, and an unfinished provisional creation still did not count. > - Those failures became a generic `setup_failed` with no bounded retry. And when the bounded retries of any `environment_provider_unavailable` failure were used up, recovery still put the task on a board-owned `stranded_assigned_issue` action. > - A hosted instance saw this for six runs in 45 minutes when concurrent runs rose from about 8 to about 20. Two tasks stayed blocked and waited for the board. > - This pull request classifies every lease acquisition failure that has structured transient evidence as `environment_provider_unavailable`. When the bounded retries are used up, the task rests on a system-owned action and a working Retry. > - It also adds the restart-chaos fault EP1, so the harness covers lease acquisition failures from now on. > - The benefit is that a busy or slow sandbox provider no longer strands tasks on the board. ## Linked Issues or Issue Description No public issue exists. **What happened?** Five runs failed with `Failed to acquire lease for environment "Paperclip Sandbox" (sandbox): Daytona lease acquisition timed out; allocation cleanup is pending` after 5 to 6 minutes in `preparing`. One run failed with `Database query failed (lease_acquire_failed)` within 10 seconds. Every run recorded `executionRecovery: { kind: "bootstrap", providerWorkStarted: false }`, but the error code was `setup_failed`. No run got a bounded retry. Two tasks were left `blocked` with a board-owned `stranded_assigned_issue` recovery action. **Expected behavior** A lease acquisition failure happens before any provider work. The run must retry with a bounded, visible backoff. When the bound is used up, the task must rest with a Retry that works, and no board-owned recovery action. **Steps to reproduce** 1. Make lease acquisition fail with the provider's acquisition deadline (the validated acquisition diagnostic that the environment runtime records), or with a database driver error. 2. Let the run start. 3. Before this change, the run fails with `setup_failed`, no retry is scheduled, and recovery creates a board-owned `stranded_assigned_issue` action. ## What Changed **How this PR and #20828 divide the problem.** #20828 (merged) makes the Daytona plugin end a stuck create early and classifies the acquisition-deadline envelope as `environment_provider_unavailable`. This PR does not change the Daytona plugin or that deadline path. It adds only what #20828 leaves out: - a lease-acquisition failure that is not a create timeout: the plugin RPC deadline (`-32003`) and an unfinished provisional creation recorded for this run; - a transient database error while the lease is recorded; - the exhausted state: a system-owned recovery action and Retry instead of a board-owned `stranded_assigned_issue`, for every `environment_provider_unavailable` cause (including #20828's); - the chaos fault EP1 (EP1, EP1N, EP1D). - `readTransientLeaseAcquisitionFailure` (`server/src/services/environment-provider-unavailable-error.ts`) classifies a lease acquisition failure as transient when it has structured evidence: the provider status or errno (as before), the run's own validated acquisition diagnostic (deadline or unfinished provisional creation), the plugin RPC deadline, or a transient database error (a postgres.js connection code, or SQLSTATE class 08 or 53, 57P01-57P03, 57014, 55P03, 40001, 40P01). A deterministic SQL error (syntax, schema, constraint) is not retried. A definite non-transient status (for example HTTP 401 or 404) is still `setup_failed`. Message text is not parsed. - `environment-run-orchestrator.ts` uses this classifier, so these failures get `environment_provider_unavailable` and the existing bounded bootstrap retry (30 s, then 60 s, with jitter). The task shows "retry scheduled" while it waits. - When the retries are used up, recovery uses the cause `environment_provider_unavailable` with a system-owned action and no board routing. The notice says "Retry when ready" and does not ask for a board decision. The task's Retry starts a new acquisition. - Restart-chaos fault EP1: a test-mode server seam (`environment-lease-fault-injection.ts`, read from `PAPERCLIP_CHAOS_LEASE_FAULT_CONTROL`, ignored in production) throws the staging error shapes before any provider call. `tests/restart-chaos/env-provider-wave.ts` runs EP1 (deadline twice, then success), EP1N (deadline forever, then Retry after disarm) and EP1D (database error once). The oracle fails on any `setup_failed` run, any board-owned recovery action, a wrong retry count, or a task that does not complete. - Docs: `doc/execution-semantics.md` and `tests/restart-chaos/README.md`. ## Verification - RED on `origin/master` (`1d67e309e0`, before #20828): the three new tests in `heartbeat-process-recovery.test.ts` ("transient sandbox provider refusal during lease acquisition") fail with the staging result: `expected [ [ 'failed', 'setup_failed' ] ] to deeply equal [ [ 'failed', 'environment_provider_unavailable' ], [ 'scheduled_retry', null ] ]`. On current master the deadline now retries (#20828), but the exhausted case still rests on a board-owned `stranded_assigned_issue` action (the existing HTTP 502 exhaustion test showed `ownerType: "board"` before the recovery change), and the database error is still `setup_failed`. - GREEN: `pnpm --filter @paperclipai/server exec vitest run src/__tests__/heartbeat-process-recovery.test.ts -t "lease acquisition"` passes 5 of 5. This includes the deadline and the database error that retry once and then complete, and the deadline that never clears. That task rests `blocked` with a system-owned action, `canRetry: true`, no execution blocker, and Retry completes it. - Unit tests: `environment-provider-unavailable-error.test.ts` + `environment-lease-fault-injection.test.ts` (56), `tests/restart-chaos/env-provider-faults.test.ts` (6) pass. - Related suites pass: `environment-run-orchestrator`, `task-interruption-query`, `issue-recovery-actions`, `issue-recovery-deadlines`, `stranded-notice`, `heartbeat-native-resume-lease-failure`. - `server` and `tests/restart-chaos` typecheck pass. - Chaos waves (local, Codex `gpt-6.1-sol`, head `1bd387e98f`; the later head changes only database classification and the notice), all pass: - EP1 `w-20261111T171803-s71-ep`, 2 tasks: each had exactly 2 `environment_provider_unavailable` runs, then succeeded with no user action, no board-owned recovery action, brief oracle pass. - EP1D `w-20261111T172850-s73-ep`, 1 task: 1 failed run on the database error, then succeeded. - EP1N `w-20261111T173346-s74-ep` (and `w-20261111T162752-s72-ep` before the rebase): 3 failed runs, the task rested `blocked` with `canRetry: true` and no board-owned action; after disarm, Retry completed it. - No Daytona wave was run. The fault is injected in the server before the provider call, so a local instance reaches the same retry and recovery code. ## Risks - More failures now retry, all bounded. Only transient SQLSTATEs retry. The staging database error recorded no SQLSTATE. If it was a deterministic SQL error, it still fails `setup_failed` and goes to the board as before; that is intended, because a retry cannot fix it. - Each retry is a new creation attempt with its own creation record. A provisional allocation whose cleanup is pending stays with its record and the creation reconciler, as before. - The retry bound is the shared bootstrap bound (two retries). A provider outage longer than about 15 minutes still rests on Retry, but no longer on a board action. - The fault seam reads a file only when the environment variable is set, and never when `NODE_ENV=production`. - Saved state from the previous release: no persisted shape changes. A task already stranded on a board action keeps it (`preserveExistingOwner`). New failures use the new cause. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude (Anthropic) through Claude Code, model ID `claude-opus-5-5`, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
Nicky LeachandPaperclip
|
b2edea5754 |
fix(workspaces): verify existing worktrees before fresh retries (#20876)
## Thinking Path > - Paperclip manages work for AI agents. > - Retry checks whether the task can keep its existing workspace. > - A provider can complete checkout before Core receives the result. > - Recreating a completed worktree costs time and can change its base commit. > - Core uses the same ownership and writer checks for local and provider workspaces. > - The provider verifies its checkout before Core creates a fresh workspace. ## Linked Issues or Issue Description **What happened?** A retry can allocate a new provider worktree even though the original checkout completed. **Steps to reproduce** Complete workspace checkout through a storage provider. Fail Runner startup or lose the prepare reply. Retry the task. **Expected behavior** Reuse the same worktree when the provider confirms checkout and the prior run can no longer write to it. Related: #20865 addresses timeouts while the original prepare is in progress. This PR checks an earlier checkout when the user retries. ## What Changed - Check the prior task binding and writer state before asking a storage provider to verify a checkout. - Extend the existing prepare operation with an optional verification mode. Providers opt in through their manifest. - Return through normal workspace reuse after verification. Keep the fresh retry fallback when verification cannot establish checkout. ## Verification - Core workspace preparation, plugin workspace ownership, retry continuity, and SDK workspace-provider suites passed locally (213 tests). - Focused cases pass for provider verification, retry through heartbeat dispatch, and real local Git checkout verification with no plugin worker. The heartbeat test confirms that ordinary preparation and adapter dispatch keep the original workspace, branch, and base commit. - Core server typecheck passed before the final audit and timing edits. GitHub CI covers the final revision and broader regressions. ## Risks Provider verification adds one bounded call before a fresh retry. Older providers keep their current behavior. Verification must preserve the workspace, branch, and recorded base commit. The normal execution preparation checks still run before admission. No database migration is needed. ## Model Used OpenAI GPT-6 (Codex), with source inspection and local test execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either linked existing issues or described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal issue details - [x] I have run targeted tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
a52f622be8 |
fix(native): retry a remote provider startup failure once its runner exit is proven (#20874)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on a remote sandbox (Daytona) start a Runner in the
sandbox, and the Runner opens the provider session
> - When the provider rejects `session.open` at startup, no provider
work began
> - #20808 ends every such remote rejection at once with Retry, because
restart recovery could not prove that the failed runner was gone. A
fault that clears by itself therefore needs one human click
> - The Runner transport already settles the failed runner on close and
observes its exit. That is the missing proof
> - This pull request records that proof on the failure and lets the
same-run bootstrap retry use it. It stops a startup error that repeats,
so an image fault that a retry cannot fix does not loop
> - The benefit is that a transient remote startup failure recovers
without a person, and a permanent one still rests at once with a working
Retry that names the error
## Linked Issues or Issue Description
Refs #20808
Refs #20832
Refs #20820
**What happened?**
A remote (Daytona) native run whose provider rejected `session.open` at
startup ended as `native_provider_startup_rejected` with Retry. This
happened even when the Runner had settled and exited, and no provider
session, event or checkpoint existed. A person had to click Retry for a
fault that the durable record proves did no provider work.
**Expected behavior**
With a proven runner exit and no provider work, the run retries its
bootstrap on the same run automatically. A startup error that repeats on
the next attempt stops at once with Retry and names that error. Failures
saved without the proof keep today's behaviour.
**Steps to reproduce**
1. `cd server && npx vitest run
src/services/native-runtime/native-startup-rejected-retry.integration.test.ts
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts
-t "remote startup rejection|session.open rejection"`
2. On master, 3 cases fail: the proven remote rejection is
`terminal_failure`, the repeat does not name its error, and restart
recovery holds the proven failure on
`ambiguous_remote_recovery_evidence`.
**Paperclip version or commit**
master at
|
||
|
Nicky LeachandClaude Opus 5.5
|
14cb69245f |
fix(projects): do not offer a single project storage option (#20875)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A project uses one workspace storage owner. The user chooses it when they create the project. > - The **Available in** selector shows the storage owners that the user can choose. > - Sometimes the default environment cannot be used, and only one storage owner is available. In that case, the dialog shows a selector with only one option. The user must select it before they can create the project. > - A selector with only one option is not a real choice. > - This pull request uses the only available storage owner as the default. The selector appears only when the user has two or more options. > - The benefit is that project creation asks for a choice only when there is one. ## Linked Issues or Issue Description Refs #20835 **What happened?** An example is an instance in managed-sandbox-only mode that has the agent-computer plugin but no managed sandbox. The local environment is hidden, and the managed-only rule finds no default. The project-environments endpoint returns one storage group and an error. The new project dialog shows **Available in** with only one option and disables **Create project** until the user selects it. An API request without `environmentId` fails with `project_environment_unavailable`. **What did you expect?** When only one storage owner is available, the project uses it. No selector appears. **Steps to reproduce** 1. Turn on `enableManagedSandboxOnly`. Install the agent-computer plugin. Do not add a sandbox environment. 2. Open **Create project**. 3. See an **Available in** selector with one option, and a disabled **Create project** button. ## What Changed - `projectEnvironmentOptions`: when the configured default is unavailable and only one storage group exists, the first environment of that group becomes `defaultEnvironmentId`, and `error` is null. When two or more groups exist, an unavailable default still produces the error. The server does not guess. - Project create uses the same default, so an API request without `environmentId` now succeeds in the single-group case. - `NewProjectDialog` shows the selector only when `groups.length > 1`. - Tests: - A new server test covers managed-only mode with only the agent-computer environment. - A new server test covers an unavailable default with two storage groups (the server still refuses to guess). - The archived-default test now expects the server to use the only remaining storage. - A new UI test checks that the dialog never shows one option. - Docs: `doc/project-environment-compatibility.md` and `docs/api/goals-and-projects.md`. ## Verification - `pnpm vitest run src/__tests__/project-environments.test.ts` in `server/`: 16 passed. - `pnpm vitest run src/components/NewProjectDialog` in `ui/`: 12 passed. - `tsc --noEmit` in `ui/`: clean. - Server `tsc` reports 9 errors on this machine. None are in the changed files. They are in `packages/adapter-utils` and a native-runtime test, because local dependencies are stale. CI typechecks with a fresh install. ## Risks - Low risk. The change applies only when the default is unavailable and exactly one storage group exists. In that case, the server picks the group that the user would have had to choose anyway. Run placement does not depend on the project, so this does not change where runs execute. - No migration. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
cda23f08ac |
fix(db): preserve legacy project reads during rolling upgrades (#20860)
## Thinking Path > - Paperclip controls agent work while web servers change versions. > - The old and new servers can share a database during that change. > - Older servers still read and write the project environment column. > - Migration 0466 removes that column before those servers stop serving. > - A later repair alone leaves a committed missing-column window and loses values. > - This pull request applies the exact drop and repair in one transaction. > - Old reads and saved values survive while current placement behavior stays unchanged. ## Linked Issues or Issue Description Refs #20835 and #20852. **What happened?** Migration 0466 drops `projects.environment_id`. The previous server still selects that column and reads it during admission. Applying a separate repair afterward leaves a window in which those requests fail. The drop also removes saved values. **Expected behavior** A database that has not yet applied 0466 must keep the column and its values visible to concurrent old servers throughout the upgrade. Both published migration entries must be recorded truthfully. Current project responses must keep the new workspace placement contract. **Steps to reproduce** 1. Use a database before migration 0466 with nonempty project environment values. 2. Keep an old project writer transaction open and start the pending migrator. 3. Start an old project read, then commit the writer. 4. Without this repair, the saved values disappear. With this repair, the reader sees the preserved values and both migrations commit together. ## What Changed - Add migration 0467 to restore the legacy nullable UUID column and its original foreign key. Leave the published 0466 SQL unchanged. - Apply only the exact SHA-bound 0466/0467 pair in one transaction. Lock the project table, retain nonempty values in a temporary table, execute both real SQL files, restore values, and record both real journal entries before commit. - Apply the additive 0467 repair normally when 0466 was already committed. - Keep a deprecated database-only field and omit it from current project API responses. Current placement logic stays unchanged. - Add PostgreSQL regressions for concurrent old reads and writes, idempotence, rollback, and an already-applied drop. Document the overlap contract and retain the five newest generated snapshots. ## Verification - Node 24 cached agent preflight passed. - `pnpm exec vitest run packages/db/src/project-environment-overlap-migration.test.ts server/src/__tests__/project-environments.test.ts -t 'project environment rolling migration|saves storage, not'`: 4 passed in 32.57 seconds against real disposable PostgreSQL. - Before the owner fix, the regression had two failures: saved values disappeared and the restore-failure case committed. The fixed case proves that both DDL and journal entries roll back together. - `pnpm --filter @paperclipai/db typecheck` and `pnpm --filter @paperclipai/db check:migrations`: passed. - `git diff --check`: passed. Migration 0466 bytes are unchanged. The generated snapshot set is 0463 through 0467. - Review found a raw DELETE response that still exposed the database-only field. The delete projection now omits it; a real authorized HTTP DELETE passed with successful cleanup. The concurrency test now observes the old reader waiting inside PostgreSQL before releasing the old writer. All three migration cases passed with that barrier. Earlier HTTP fixture setup/cleanup failures remain recorded as failed attempts; they were not product-regression passes. - Candidate `047178538c0df3a9e382259a51635a6e46da5646` has fresh Greptile 5/5 with no open findings. [Required final-head CI](https://github.com/paperclipai/paperclip/actions/runs/43062354612) passed, including build, typecheck, migration checks, all test shards and the aggregate verification gate. The separate Superagent security scan ended inconclusive/neutral; it did not produce a passed scan. Deployed acceptance remains pending. No hosted database or app deployment was changed by this branch. ## Risks The pair takes a table lock for its transaction. Operators must retain their normal lock and statement timeouts. Both SQL files and their exact hashes must match before the pair runs. A failed restore rolls back the drop and both journal entries. If 0466 was already committed, 0467 restores the column and foreign key. It cannot recover values that were already deleted by the earlier release. This repair does not claim to recover those values. ## Model Used OpenAI Codex (GPT-6), with tool-assisted implementation and real PostgreSQL qualification. The exact serving model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
e0e6462a73 |
fix(heartbeat): keep an agent's explicit resume comment that a stale continuation dropped (#20856)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agents coordinate on tasks through comments. A comment posted with `resume: true` is an explicit request for more work; on a finished task it reopens the task, for a person or for another agent. > - A comment that arrives while the assignee's run is still settling queues its own run. If the settling run then closes the task, that queued run is cancelled before dispatch as a stale continuation (`continuation_task_ownership_changed`). > - For a person's comment, the outbox saves the unread message first and the release drain reopens the task (chaos s9401/t72). For another agent's explicit `resume: true` comment, nothing was saved. The request was dropped and the task stayed `done`. > - This pull request saves that agent comment too, with its explicit resume intent. The existing release-drain rule for an agent's explicit resume reopens the task and delivers the comment once. > - The benefit is that a parent agent's "do this next turn" request is not lost to a timing race. An agent comment without `resume: true` stays communicative, as before. ## Linked Issues or Issue Description **What happened?** On a hosted staging instance, a parent agent posted a comment with `resume: true` on its child's task: "a changed requirement for your NEXT turn". The child's provider turn had finished 90 s earlier, but its run was still settling (workspace copy-back). The comment queued a new run. 40 s later the settling run applied its `done` decision. The queued run was then cancelled with `continuation_task_ownership_changed`. The task stayed `done` and no run ever read the comment. The cancelled run itself is not shown as a failure (the task is terminal, so task chat hides it, and Sentry skips expected cancellations), so nothing told anyone the request was lost. **Steps to reproduce** 1. Run an agent on a task. While its run is still settling, post a comment from another agent with `resume: true`. 2. Let the run close the task (`done`). 3. The comment's queued run is cancelled as a stale continuation. The task is not reopened and the comment is never delivered. **Expected behavior** The task reopens for the explicit resume request, and the comment is delivered exactly once, as for a person's comment. ## What Changed - `deferUndeliveredCommentsOfStaleRun` finds, per owed comment, the comment's own `issue.comment_added` activity record: written by its author (another agent, not the assignee) with `resumeIntent: true`. Merged runs, whose wake is rewritten as comments join, therefore keep each comment's own intent. - `saveUndeliveredUserComments` also admits those comments (`explicitResumeCommentIds`). When only such comments remain, the deferred receipt records the agent as requester and carries `resumeIntent`/`followUpRequested` in its payload and wake context. The release drain's existing rule for an agent's explicit resume on a `done` task then reopens and delivers it. Person-comment behaviour is unchanged. - `doc/run-log-events.md` describes the rule. ## Verification - RED on `origin/master` (`1d67e309e0`): `pnpm --filter @paperclipai/server exec vitest run src/__tests__/heartbeat-held-comment-wake.test.ts -t "explicit resume comment"` fails: `expected [] to have a length of 1 but got +0` (the comment is never delivered after the stale cancellation). - GREEN: the whole file passes (29/29), including two-requester merged-run tests in both orders, the new negative test (`does not reopen a finished task for another agent's comment without resume intent`) and the existing person-comment and exactly-once tests. - Related suites `ownership-reconciliation-report`, `execution-continuation`, `chat-wake-conversation-batch` plus this file: 118/118 tests pass locally. One `afterAll` cleanup hook in `ownership-reconciliation-report` timed out under local machine load; CI runs the suite fresh. - `pnpm --filter @paperclipai/server exec tsc --noEmit` passes. - Not done: no restart-chaos cell. The harness posts follow-ups as the board user, which s9401/t72 already covers; an agent-authored `resume: true` comment needs a two-agent workload the harness does not have. ## Risks - Low risk. Only a comment whose own wake was an agent's explicit `resume: true` qualifies, only on a task an agent run closed, and only when no run read the comment. The assignee's own comments never qualify. - The ownership reconciler's durable backstop (`checkStaleDroppedComments`) still covers person comments only. If the fast path fails, an agent's resume comment is not recovered later. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, read-only staging API reads). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
e6b538efd2 |
fix(native): retire a superseded warm owner instead of failing the new run (#20844)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs keep a warm Runner between turns of the same task, so
the next run can start fast
> - When the next run needs a new provider session (for example after an
agent setting or skill changes), it first retires the previous run's
warm Runner
> - That retirement uses the previous owner's close. When the close
cannot prove checkpoint suspension, the new run failed before any
provider work, with the previous owner's error
`provider_transport_failed: runner did not durably suspend before
checkpoint`
> - A governed-wait park (a saved question or review request) always
leaves such an owner: its question callback stays unsettled by design. A
failed Launcher stop RPC leaves one too
> - Nothing retried the run, and the task rested as "recovery needed"
with no provider work done
> - This pull request lets the new run retire the old owner when that
owner is contained, and forces the containment when the close left it
unproven
> - The benefit is that a fresh run never fails on its predecessor's
close, and the chaos harness now covers this class
## Linked Issues or Issue Description
**What happened?**
A new run on a task whose previous run left a warm Runner retained
failed within two minutes, before any provider turn. The error was
`provider_transport_failed: runner did not durably suspend before
checkpoint`. The run row recorded `executionRecovery: { kind:
"bootstrap", providerWorkStarted: false }`. Two runs of the same agent
on two tasks failed this way 47 seconds apart, after a configuration
change made both start new provider sessions. One predecessor had parked
on a `request_human_input` question; its callback never settled.
**Expected behavior**
A run that has done no provider work does not fail because its
predecessor's Runner could not prove suspension. The new session never
resumes the old checkpoint. Core must only prove that the old owner is
contained, and then start.
**Steps to reproduce**
1. Run a warm native task that parks on a saved question (chaos workload
Q1).
2. After the asking run settles, change the agent's adapter
configuration, so the next run cannot resume the saved session.
3. Answer the question. The follow-up run retires the parked owner and
fails with the error above.
Related: #18373 added the stopped-owner reset path that this change
extends.
## What Changed
- `closeWarmNativeSession` (server native executor): during retirement
for a different normalized session, accept an unrecoverable close when
the old owner is contained. A plugin task owner must show its
`executionStopped: true` receipt and a completed handoff. Without a task
owner, the Runner's own durable suspension
(`settlement.runnerSuspended`) is the proof. Unsettled Core tool
callbacks of the discarded conversation no longer block the reset.
- With a plugin task owner, any close error qualifies once the Launcher
receipt exists. This includes a failed `stop` that
`releaseRunnerProcessOwnership` rethrows, and "Controller journal drain
timed out": the discarded session's journal lives in its own state root.
Owners without a task keep the narrower rule. `--matrix` now rejects
session faults.
- `createPluginRunnerTaskOwner`: add `retireSupersededTask()`. When
close left the receipt unproven, the new run sends one confirmed `stop`
(not resent after a confirmed stop), takes the credential snapshot, and
calls `complete`. It never reads the old state.
- Same-session continuation, a handoff that still fails, other close
errors and failed instruction collection still block replacement.
- Chaos harness: add session fault SR1
(`tests/restart-chaos/session-faults.ts`). With Q1 and `--lifecycle
warm` (or `--daytona-warm`), the board changes
`adapterConfig.idleTimeoutMs` after the asking run settles and before it
answers. `--tasks 2 --concurrency 2` is the two-concurrent-runs variant.
Q1's oracle applies, plus `session_rotated_before_answer` and
`follow_up_new_session`.
- Docs: `doc/plugins/PLUGIN_RUNNER_EXECUTION.md` and the chaos README
describe the new boundary and cell.
## Verification
- RED on master: `native-session-executor.test.ts -t "permanently
rejected plugin close during reset"` fails the `pending-semantic` and
`stop-unconfirmed-at-close` cases with
`NativeSessionCloseUnrecoverableError: provider_transport_failed: runner
did not durably suspend before checkpoint`. GREEN with this change: all
identity-transition cases pass (49).
- Chaos SR1 local, master server code (`--env local --lifecycle warm
--workloads Q1 --faults SR1`, seeds 301 with 2 tasks and 303 with 1
task): all tasks pass, and the follow-up started a new provider session.
The local parked owner closes cleanly, so the local cell does not
reproduce the failure. It needs a Daytona task owner.
- Chaos SR1 local with this change (seed 302, 2 concurrent tasks on one
agent): 2/2 pass, `follow_up_new_session` true for both.
- Not run: the Daytona SR1 variant. No runner image matches this
checkout's runner content ID, and this change does not touch runner
inputs.
- `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts
question-handoff scheduler config` passes. Server and harness `tsc
--noEmit` pass.
## Risks
- The reset no longer waits for unsettled Core tool callbacks of the
discarded session. Their run is already terminal and their provider is
stopped, so no result can reach a provider, and the new session never
replays them. Core-side tool executions keep their own receipts under
the old run.
- A local owner without a task needs the Runner's durable suspension. A
local owner that did not suspend still blocks replacement, as before.
- Previous-release story: this changes no persisted state. Warm owners
live only in controller memory, so a deploy drops them as before.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, agentic coding with
tool use (shell, file edits, test runs) in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandClaude Fable 5.2
|
464e1de9b1 |
perf(issues): decide task reads for a list in one batch (#20858)
The task routes that filter several rows for one actor (summaries, lists, blockers, relation edges and the chat bootstrap) asked for one authorization decision per task, all at the same time. Each decision sent a task-row read and a copy of the task-privacy statement, which is more than 1 KB of SQL, so a request for 50 summaries sent 127 to 279 statements on every pool connection at once. The routes now ask the authorization service for all decisions in one batched read. It reads the task rows and applies the same privacy predicate to all of them in one statement, and reads membership, agent and run policy once. Within a batch, the authorizing user's membership is read once, and the boundary checks of a low-trust agent, which stay per task, run at most eight at a time. A request for 50 summaries now sends 22 to 28 statements (65 for a low-trust agent). Tests compare the batch with one check per task for 24 actors. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
DottaandClaude
|
c8ae02e5ce |
fix(native): start a fresh session when an accepted run's provider state was not saved (#20811)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native run in a per-turn remote sandbox saves a provider state backup after its result is accepted. The sandbox is then released, so the backup is the only copy of the provider session. > - When that backup fails to save (for example, #20772: storage throttling), the run keeps its accepted result. Nothing records that its checkpoint can no longer be resumed. > - The next run on the session rebinds that checkpoint and asks the Runner to resume. The Runner has no state and refuses (#20757). With the normal policy, the run opens a replacement session after that failed attempt. With a governed-wait or unfinished-goal checkpoint, the run fails with `native_provider_state_incompatible`, and only Retry or the next message starts fresh. > - The server already knew the state was gone when the backup failed. The failed resume is wasted work. > - This pull request marks the accepted run's checkpoint as not resumable at the moment the backup fails. The next run then starts a fresh provider session directly. > - The benefit is that a lost backup no longer costs a failed run or a failed resume attempt before the task continues. ## Linked Issues or Issue Description Refs #20757 (the missing-state refusal that the next run reaches today). Refs #20772 (the throttled backup upload that loses the backup). Refs #20809 (a related fix for governed-wait follow-ups that reach the missing state; this PR stops most of them from reaching it). **What happened?** 1. A native run in a per-turn sandbox finished, and its result was accepted. 2. `harness_state.backup.persist` failed. The run recorded `native.post_result_failure`. Its run row kept a normal `sessionCheckpoint`. 3. The sandbox was released. 4. The next run on the same session selected that run as its resume source, and `rebindNativeSessionCheckpoint` returned the checkpoint. 5. The Runner found no saved state and refused with `PRP provider resume state is unavailable`. The run then opened a replacement session after the failed attempt, or (governed-wait or unfinished-goal checkpoint) failed with `native_provider_state_incompatible`. Retry or the next message then started fresh. **Expected behavior** The next run starts a fresh provider session at once, because the server already knows the saved state is not there. The accepted result and the task outcome stay as they are. **Steps to reproduce** 1. Give a run row an accepted result and a normal `sessionCheckpoint`. 2. Make its close checkpoint fail in a per-turn sandbox (`reusableLeaseConfigured: false`). 3. Select the next run's resume source with `findNativeSessionResumeRun` and call `rebindNativeSessionCheckpoint`. 4. On `master` it returns the checkpoint, so the next run tries to resume state that does not exist. **Paperclip version or commit** `master` at `556b249897`. ## What Changed - `server/src/services/native-runtime/native-provider-state-unsaved.ts` (new): `markAcceptedRunProviderStateUnsaved` adds `nativeProviderStateUnsaved: { code: "native_provider_state_incompatible", reason: "harness_backup_not_saved" }` to the run's `runnerProfileJson`. One conditional `UPDATE` writes it only when the run's coordinator holds a result with `schema_status = 'accepted'`. `clearRunProviderStateUnsaved` removes it. - `server/src/services/native-runtime/native-session-executor.ts`: the remote close checkpoint (`checkpointRemoteRunner`) now wraps the existing body. When the checkpoint throws in a sandbox without a reusable lease, it calls the marker, writes one run-log line, and throws the original error. A marker or log failure cannot replace that error. If a later checkpoint of the same run saves, it clears the mark. - `server/src/services/native-runtime/native-session-resume.ts`: `rebindNativeSessionCheckpoint` returns `null` for a source with the mark, as it already does for `errorCode = native_provider_state_incompatible`. `findNativeSessionResumeRun` projects the new key. - `doc/execution-semantics.md`: one additive paragraph. - Tests in `native-session-resume.test.ts` and `native-session-executor.test.ts`. Guard conditions. The mark is written only when all of these are true: - The close checkpoint threw before the durable backup was stored (after the upload's own bounded retries). A later local publication, stamp or cleanup failure does not mark the run. - The target is a remote `sandbox` with `reusableLeaseConfigured !== true`. A retained sandbox or an SSH host keeps the live state, so its resume path is unchanged. - The run holds an accepted result. A failure before acceptance changes nothing. Saved state from the previous release: run rows without the key keep today's resume path. An older server that reads a row with the key ignores it, and so it behaves as it does today. No schema change. User-visible: the next run's transcript shows "Paperclip session started" instead of "resumed", with a new session id. The source run's log shows `provider state was not saved after the accepted result; the next run starts a fresh provider session`. The fresh session gets the full task context, as for other fresh starts. ## Verification - `cd server && npx vitest run src/services/native-runtime/native-session-resume.test.ts -t "not resumable only after|provider state was not saved"`: 9 pass (6 pure, 3 embedded Postgres). - `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts -t "marks the provider state unsaved|uses the bundled Linux image authority for remote opencode-new-durable"`: 8 pass. They include a durable store failure (marked) and a local publication failure after the store (not marked). - Red on `master` (source files reverted, tests kept): the 3 "starts the next run in a fresh provider session directly" cases fail (`expected {…} to be null`), the DB "accepted" case fails the same way, and the executor per-turn case fails (`mark` called 0 times). - Guard removal checks: - Sandbox guard replaced with `true`: the "retained sandbox" case fails (`mark` called once). - `!checkpointBackupStored` removed: the "publish-failed" case fails. - `schema_status = 'accepted'` removed from the marker: the "rejected" case fails (`expected true to be false`). - Negative cases that pass: no result, rejected result, retained sandbox, records without the key or with another code (all three recovery policies), and a later save that clears the mark. - `cd server && npx tsc --noEmit -p .`: clean. - Not related to this change and failing on `master` in this worktree too: `runtime-selection.test.ts` connector skill cases (missing `.claude/skills/*/SKILL.md`) and `native-session-resume.test.ts` "archives a damaged prior epoch" (`ENOENT … authority-epochs`). Required CI is the authority for those suites. ## Risks - Low. The change only skips a resume that cannot succeed. The fresh session uses the same workspace and full task context, like a Retry after `native_provider_state_incompatible`. - If the backup failed but the provider state were still somewhere recoverable, the next run would lose provider context. For a per-turn sandbox without a reusable lease, the restore path uses only the backup, so this does not apply. Retained sandboxes and SSH hosts are excluded. - One extra conditional `UPDATE` runs only on the failure path. The normal path before `turn.start` has no new waits or work. - `doc/execution-semantics.md` is also edited by #20809. The paragraphs are at different places; whichever merges second rebases. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code, with tool use (shell, file edit, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
1b39cf1794 |
fix(native): retry a contended provider-pack admission write (#20850)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native OpenCode run on a remote sandbox must save a provider-pack admission receipt on its run row after the provider session opens. Artifact retention refuses to continue without it. > - The admission write locks the coordinator row (`native_run_finalizations`) and then the run row, each with a 1000 ms `lock_timeout`. The runner's durable checkpoint writes lock the same coordinator row for the whole checkpoint transaction. > - On a loaded database one checkpoint transaction can outlast one lock wait. The admission write then fails with `55P03`, the state stays `pending`, and artifact retention fails the run as `native_provider_artifacts_admission_unverified` before the first turn. The task rests blocked. > - #20682 fixed the same symptom for a run-event append on the run row. This is the coordinator-row half. > - This pull request retries a contended admission write a bounded number of times, each in a new transaction that re-proves the claim. It also names the database cause in the run log and adds a chaos option that makes every fresh sandbox take the pack upload and admission path. > - The benefit is that a healthy fresh sandbox is no longer failed and blocked by ordinary lock contention. ## Linked Issues or Issue Description Refs #20682 (same symptom, run-row lock; this is the coordinator-row lock). **What happened?** On a hosted staging instance, an OpenCode run on a fresh Daytona sandbox found no matching preinstalled provider pack (`provider_pack.select` reason `cached_packs_unverified`). It uploaded and verified the pack (61 s), started the runner, and the runner authenticated to the durable control plane. 3.4 s later the run log shows `native provider-pack admission receipt could not be saved`. `nativeProviderPackAdmissionState.state` stayed `pending`. The run failed with `native_provider_artifacts_admission_unverified` before any provider turn. The task rests `blocked` with no Retry. **Steps to reproduce** 1. Start a native run that waits for its claim (`waitForClaim: true`, as a fresh run with artifact storage does). 2. Hold its `native_run_finalizations` row `FOR UPDATE` in another transaction for 1.6 s, as a checkpoint write does. 3. Call `recordNativeProviderPackAdmission`. It fails with `canceling statement due to lock timeout` (`55P03`). **Expected behavior** The admission waits out the checkpoint write within a bound, re-proves the claim, and saves the receipt. ## What Changed - `recordNativeProviderPackAdmission` retries a write that waits for its claim when it fails with `55P03`, `40P01` or `40001`, up to 5 attempts with 100–400 ms back-off. Each attempt is a new transaction with the same checks (claim, run, epoch, receipt). A stale claim still returns `false`. Other errors still throw. A write that does not wait for its claim keeps its `NOWAIT` refusal. - New `nativeProviderPackAdmissionFailureCode` reads the SQLSTATE from the error or its cause. - The executor's run-log line now ends with `(cause=<SQLSTATE or native_* code>)`, so a lost admission is diagnosable. It carries no paths or secrets. - Restart chaos: `--daytona-pack-mismatch <image@sha256:…>` starts sandboxes from another image while the controller keeps `--daytona-image`'s runnerd and pack. Every fresh sandbox then takes the upload and admission path, which the matched-image check made unreachable. Config validation and README updated. - `doc/execution-semantics.md` describes the bounded retry. ## Verification - RED on `origin/master` (`1d67e309e0`): `pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/native-provider-pack-admission.test.ts -t "checkpoint write holds"` fails with `canceling statement due to lock timeout` (`55P03`), the staging error. - GREEN: the same file, 39/39 pass, including the existing fence tests (`cannot publish while another transaction holds the coordinator claim`, revoked-claim recheck) and the new bound test (`gives up after the bounded retry window while the claim stays held`, under 15 s). - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts tests/restart-chaos/scheduler.test.ts`: 34/34 pass, including the new option test. - `pnpm --filter @paperclipai/server exec tsc --noEmit` and `tsc --noEmit -p tests/restart-chaos/tsconfig.json` pass. - Not run: a Daytona chaos wave with `--daytona-pack-mismatch`. It needs a runner image built at this revision. The staging cause is inferred from the timing and the code path; the original error was swallowed, which is why this PR adds the cause to the run log. ## Risks - Low risk. The normal path has no contention and makes one attempt. Under contention the write waits at most about 11 s before the first provider turn. Without this change that case failed the run. - A retry cannot widen authority: every attempt re-checks the coordinator claim, lease expiry, run status and admission epoch. - When admission still fails after the retries, the run fails as before. This PR does not change that recovery path. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, read-only staging API reads). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Fable 5.2
|
8ab955776d |
perf(github): check the requester provenance of an issue list's review waits in one batch (#20851)
<!-- Write all pull request text in Simplified Technical English (ASD-STE100): short sentences, one instruction per sentence, simple approved vocabulary, and the active voice. --> ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - An agent can review a GitHub pull request. After the agent publishes the review, the task waits in review for the next GitHub reply. A later run of the same task can inherit this wait. > - Issue lists, issue summaries and inbox attention show these waits. For each waiting task, the server first checks the requester provenance of the waiting run (`chatRunSourceMessageId`). This check is an authorization check: it walks the identity-context lineage of the run and rejects a changed requester, another task, another company or another provider. > - #20673 batched the review reads and the publication reads of one list request. It kept the provenance check as one call for each waiting run, inside `Promise.all`. > - Thus a request with N waits still built and sent at least 2N statements at the same moment. The database is fast for each statement, but the server builds each statement on its main thread. > - On a hosted instance, a CPU profile of the server showed the main thread at 96% of one core. SQL text building, per-statement timers, socket writes and garbage collection had the largest shares. Connection samples showed 235 to 292 pool connections whose last statement was the source-message link read of this check, and 53 to 112 whose last statement was its identity-context read. > - This pull request checks the provenance of all the waiting runs of one request in one batch. The runs that need the same kind of read share one statement. Each run gets the same decision as a separate call, from the same walk and the same reads. > - The benefit is that a list request with 250 waits sends 17 statements, not 658, and uses about one quarter of the CPU time. The statement count no longer follows the number of waits. ## Linked Issues or Issue Description Refs #20673 (batched the review reads and the publication reads of the same request; this change does the same for the provenance reads that #20673 kept for each run). Refs #20551 (authorizes the waits of one issue page with one read). **What happened** `readInheritedGitHubReviewWaits` called `chatRunSourceMessageId` one time for each waiting run, inside `Promise.all`. Each call sent an identity-context read and a source-message link read. A run that continues an earlier run also sent a parent-context read and a source-run read for each continuation in its lineage. `listExternalReviewWaits` calls this function for the issue list, the issue summaries and the inbox attention of a company, so each of those requests repeated the same reads for every waiting task. Evidence from a hosted instance (sanitized): - CPU profile of the server process, 30 s, 5 ms samples: main thread at 96% of one core. Building SQL text for select statements: 17.7% inclusive. The idle-timer restart that the database client does for each statement: 7.6%. Socket writes: 6.2%. Garbage collection: 13%. - Three samples of the pool connections (about 494 connections). 235 to 292 connections had the source-message link read of `chatRunSourceMessageId` as their last statement. 53 to 112 connections had its identity-context read as their last statement. - The `DISTINCT ON` review read that #20673 added ran about 29 times each minute. It runs one time for each list request, so each request started hundreds of provenance reads. **Expected behavior** One list request sends a number of provenance statements that does not follow the number of waiting tasks. The decision for each run does not change. **Steps to reproduce** 1. Put many tasks of one company in review, each with an inherited GitHub review wait. 2. Open the issue list or the inbox. 3. Count the statements that the server sends. The identity-context read and the source-message link read run one time for each waiting task. The benchmark below does these steps on a local PostgreSQL. ## What Changed - `server/src/services/chat-channels/run-origin.ts`: - The provenance decision is now one walk (`sourceMessageWalk`). The walk asks for one read at a time and decides on the returned rows only. Its checks, their order and its error messages are the checks of the previous `chatRunSourceMessageId`. - The five reads of the walk (active identity context, source-message link, board message, parent context, source run) are defined one time in `provenanceReads`. Each definition takes its values either as literals or as SQL columns. - `chatRunSourceMessageId` (single run) drives the walk with one statement for each read, as before. The joins, the company scope and the filters are the same. The identity-context reads now select the seven columns that the walk uses instead of every column. - New `chatRunSourceMessageIds(db, runs, options)` (batch). It drives one walk for each run. In each round, the runs that wait for the same kind of read share one statement: `unnest(...)` of the request values, `CROSS JOIN LATERAL` the same single-row read with the request columns in place of the literals. Thus each run gets the row that its own single read returns, with `LIMIT 1` and the company scope on each join. The result for each run is `{ messageId }` or `{ forbidden }` (the 403 that the single call throws). A database error rejects the whole batch, as `Promise.all` did. - `server/src/services/chat-channels/github/review-wait.ts`: `readInheritedGitHubReviewWaits` checks the provenance of its runs with the batch. A 403 still gives `revoked`. The single-run `readInheritedGitHubReviewWait` does not change. - `server/src/services/chat-channels/run-origin.test.ts`: two new tests (see Verification). - `server/src/__tests__/github-review-wait.test.ts`: one new regression test for the statement count. - `server/scripts/benchmark-review-wait-provenance.ts`: the reproduction and the before/after measurement. It loads the pre-fix modules from `--baseline-ref` and fails unless both versions return the same waits. - `scripts/benchmark-github-review-wait.ts`: removed an assertion that `run-origin.ts` is byte-identical to its baseline. Both sides of that benchmark import the current helper, so the comparison stays fair. Without this change, the documented command of that benchmark fails. - `doc/DATABASE.md`: describes the batch provenance check and the new benchmark command. Statement count of one batch: the walk of one run does at most 3 + 4L reads, where L is the number of continuations in its requester lineage. The batch does the same number of rounds for its longest lineage. Each round sends at most one statement for each kind of read (5 kinds), for any number of runs. A request whose waiting runs accepted their GitHub comment directly sends 2 provenance statements. ## Verification Red/green regression test (`server/src/__tests__/github-review-wait.test.ts`): - `reads the waits of forty runs with the statements of four runs`. Before the fix: `expected { few: 10, many: 82 } to deeply equal { few: 4, many: 4 }`. After the fix: passes. Equivalence tests (`server/src/services/chat-channels/run-origin.test.ts`), on real PostgreSQL: - `gives every run of a batch the decision of its own check, in every provenance state`. 33 runs, each in a different provenance state, are checked with the single function and with the batch for five option sets (no options, legacy wake allowed, GitHub, GitHub with legacy wake, Slack). The test compares the decision of each run (message, `null`, or the 403 and its message) for: the full batch, the reversed batch, and a batch of one run. The states include the negative cases: a run of another company, a message that is linked only in another company, a message that is linked to another task, a conversation that moved to another task, a continuation on another task, a changed requester (run, identity context, continuation, board message), a link of another provider, a missing link, an outbound link only, a deleted message, an agent message, a rejected or pending identity context, a missing parent, another agent and a lineage cycle. The test also states the expected GitHub decision for each state, so the comparison cannot pass on two wrong answers. - `checks the runs of a batch with statements that follow the lineage, not the number of runs`. 4 runs and 40 runs each send 2 statements. 40 separate checks send 80. One continuation in each lineage gives 4 statements for the batch. - A mutation check: when the batch gives a row to the wrong request, both tests fail. Commands: - `pnpm vitest run server/src/services/chat-channels/run-origin.test.ts server/src/__tests__/github-review-wait.test.ts` - Focused run of 15 test files that import the changed modules, their callers or the shared seed helper, including `external-chat-wait.integration.test.ts`, `chat-github-tools.test.ts`, `slack-authority.test.ts` and `issue-list-review-wait-authorization.integration.test.ts`: 482 tests passed. In `idle-completed-history.test.ts`, `afterEach` hooks (a `TRUNCATE` of the test database) timed out at 30 s for 1 test in this run and for 3 tests in a second run of the file, while other agents loaded the machine. No assertion failed. The three tests of that file that reach the changed code pass when they run alone. CI runs the full file. - `server/src/__tests__/chat-channels.integration.test.ts` also uses the single check. It was not run locally (very large file). CI runs it. - Typecheck of the changed files and of the callers of `chatRunSourceMessageId` with a focused `tsc` program: no errors in these files. The full server typecheck was not run locally (shared machine). CI supplies it. - The first CI run of this pull request failed `ci / policy`: the module boundary check rejects an insert into the agents table outside the agent lifecycle module, and the benchmark seeded its fixture agent that way. The second commit seeds that agent with SQL. `pnpm check:module-boundaries` passes locally, and the benchmark gives the same statement counts. - `pnpm verify` returns exit 2 (broader verification required) because the change adds a file under `server/scripts/`. CI supplies the broad coverage. Before/after measurement: ```sh pnpm --filter @paperclipai/server exec tsx scripts/benchmark-review-wait-provenance.ts --baseline-ref=e2cf9a670fc5d5f5058fd0972beb71650f77c554 ``` One call of `listExternalReviewWaits` (the read of one issue list request). Embedded PostgreSQL 18.1, Node v26.4.0, `VACUUM ANALYZE` after seeding. One company, one agent, one GitHub repository. Every wait has a published review. Every tenth waiting run continues three earlier runs. 3 warm-up calls, then 15 pairs in alternating order. Both versions return the same waits and the same inherited wait for each run. | Waits | Version | Statements | SQL text bytes | Wall ms, median (min–max) | CPU ms, median (min–max) | | --- | --- | --- | --- | --- | --- | | 20 | before | 60 | 31,084 | 14.1 (12.8–19.6) | 13.7 (8.7–39.3) | | 20 | after | 17 | 15,847 | 12.5 (9.4–22.7) | 8.8 (4.1–87.1) | | 250 | before | 658 | 300,611 | 63.3 (58.1–89.6) | 78.1 (64.8–103.3) | | 250 | after | 17 | 17,171 | 20.8 (18.9–29.2) | 18.7 (14.1–27.3) | | 1000 | before | 2,608 | 1,179,710 | 256.0 (221.0–363.2) | 303.3 (262.2–507.8) | | 1000 | after | 17 | 21,695 | 54.9 (46.7–81.7) | 51.0 (41.6–72.2) | Of the 658 statements before the fix at 250 waits, 650 are provenance reads: 325 identity-context reads, 250 source-message link reads and 75 source-run reads. After the fix, 9 statements do the same reads. The other 8 statements of the request do not change. Sizes are set with `WAITS=20` and `WAITS=1000`. Other agents used the same machine during the measurement, so the maximum values have noise. - The batch statements use index scans only (`run_identity_contexts_pkey`, `run_identity_contexts_company_run_idx`, `chat_message_links_company_comment_idx`, `chat_conversations_pkey`, `chat_endpoints_company_id_uq`, `heartbeat_runs_pkey`). The 9 batch statements of a request with 250 waits take about 1 ms of server execution in total. - Wall time and CPU time are almost equal before the fix. This shows that the cost is in the server process, not in the database. Not verified: - The effect on the hosted instance. This pull request does not deploy. The share of the main-thread CPU that this path uses there is not known; the profile gives the share of all statement building. ## Risks - The provenance check is an authorization check. The risk is a batch decision that differs from the single decision. Mitigation: both paths run the same walk and build each read from the same definition. The batch read is the single read inside `CROSS JOIN LATERAL`, for each request row. The equivalence test compares both paths in every state. - A database error (for example an invalid UUID in a legacy task binding) rejects the whole batch. The previous `Promise.all` also rejected the whole request. The callers of the batch pass the native task of each run. - The statement count of the batch follows the longest lineage of its runs. A batch with one very long lineage sends as many statements as that run sent before, in sequence. It does not send more. - The batch sends its statements in sequence on one connection. Before, the reads of N runs used up to N pool connections at the same moment. - The single-run callers (GitHub tools, review publication, response comments, Slack authority, the finalizer, the status committer) keep one statement for each read. Their identity-context reads select fewer columns. - No migration. No new index. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude Code with tool use (code reading, editing, shell, tests). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
e625505ac9 |
fix(auth): preserve credential sign-in during server overlap (#20839)
## Thinking Path > - Paperclip controls AI agent work across software updates. > - Authenticated instances must keep human sign-in available during an update. > - Better Auth 1.7.16 identifies accounts by provider and account ID. > - It creates credential accounts without the issuer required by the previous reader. > - A new account can therefore fail to sign in through a surviving 1.7.2 process. > - This pull request keeps the legacy credential issuer on trusted account creation. > - The security upgrade and provider/account uniqueness stay in place. ## Linked Issues or Issue Description Refs #18906. **What happened?** After the Better Auth upgrade, a credential account created by the new version has a NULL issuer. A surviving Better Auth 1.7.2 process rejects its correct password because it selects only the `local:credential` account identity. **Expected behavior** A new credential account must sign in through both versions during an update. The new version must keep its provider/account lookup and uniqueness rules. **Steps to reproduce** 1. Apply migration 0465 to a disposable database. 2. Create an email/password account with the new auth mount. 3. Sign in to the same database through Better Auth 1.7.2. 4. Without this change, signup and new-version sign-in return 200, but old-version sign-in returns 401. ## What Changed - Register issuer as an optional internal account field with client input and response output disabled. - Set `local:credential` in the trusted before-create hook. Preserve any nonempty issuer supplied by internal code. - Leave unconfigured providers without an issuer on the new NULL-issuer behavior. No social provider is configured by this module. - Add credential integration and issuer-policy controls. Document the exact compatibility scope. ## Verification - Node 24: `node scripts/agent-preflight.mjs` passed using the existing build cache. - `node cli/node_modules/tsx/dist/cli.mjs packages/db/src/test-postgres-command.ts -- node node_modules/vitest/vitest.mjs run --project @paperclipai/server server/src/__tests__/better-auth.test.ts server/src/__tests__/better-auth-credential-signup.integration.test.ts`: 22 passed. - Separate local qualification used the actual cached Better Auth 1.7.2 SDK and the current 1.7.16 mount against one disposable PostgreSQL database. Before this change, old-version sign-in returned 401 for the new account. After this change, signup through either writer and sign-in through both readers returned 200. - The integration covers forged client issuer input, the frozen 1.7.2 credential selector, current-version sign-in for existing NULL-issuer rows, and the unchanged provider/account unique index. - `pnpm --filter @paperclipai/server typecheck`: passed, including all server and test-tree layers. - The first required Runner support run failed three SDK proof suites because two generated lock references were stale on the current base. The existing generator corrected those references and the three suites passed locally (58 tests in 17.07 seconds). Master then merged the same repair in #20834 and advanced its runtime/dependency lock. This branch was rebased onto current master; the duplicate proof commit was omitted. The final diff contains only the four auth/test/doc files. The exact-head checks and review listed below passed on the rebased candidate. - Final candidate `577fd7612a6d80ac35e2f5ee9ea2ce9b77169687`: focused auth verification passed again (22 tests, 51.81 seconds). [Required CI](https://github.com/paperclipai/paperclip/actions/runs/43051216288) passed, including build, typecheck, server/Runner tests and all eight browser shards. One unchanged Rust process-lineage orphan-census test failed on the first attempt; the failed job was rerun after the workflow completed and passed. This PR changes no Rust source. - [Greptile review](https://github.com/paperclipai/paperclip/pull/20839#issuecomment-6799551882) is 5/5 on the final head with no actionable findings or open review threads. No deployment or original database change was performed for this branch. ## Risks The bridge applies to new credential accounts. It does not backfill existing NULL issuers or claim old-reader compatibility for unconfigured social providers. Trusted internal issuers retain their exact value. Account lookup, password checks, account linking, provider/account uniqueness, and migration 0465 remain with their existing owners. ## Model Used OpenAI Codex (GPT-6), with tool-assisted code editing and local PostgreSQL qualification. The exact serving model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
57f1afdf19 |
fix(daytona): retry a sandbox create that timed out before any agent work (#20828)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - On a Daytona sandbox environment, every new task run first creates a fresh sandbox > - When one Daytona create did not start, the plugin waited the full 300 s acquisition budget > - The run then failed `setup_failed` with `lease_acquire_failed`. No agent work had started, but nothing retried it > - On staging this hit two agents in 15 minutes. Each attempt used about 5 minutes, and the task stopped until a person pressed Retry. The manual Retry then got a sandbox in 28 s > - This pull request ends a stuck create early, adopts a create that started late, and routes the timeout to the bounded bootstrap retry that already exists for transient provider refusals > - The benefit is that a stuck create costs about 90 s and recovers by itself. The task shows Retry only after the bounded attempts are used up ## Linked Issues or Issue Description Refs #20626 (bounded resume of a reusable sandbox), Refs #20212 (wait out a Daytona state change), Refs #20609 (force a stuck stop). **What happened?** A run on a Daytona sandbox environment failed after 307 s with `Failed to acquire lease for environment "Paperclip Sandbox" (sandbox): Daytona lease acquisition timed out; allocation cleanup is pending`. The run recorded `errorCode: setup_failed`, `failureCause: lease_acquire_failed`, and `executionRecovery: { kind: "bootstrap", providerWorkStarted: false }`. The task became blocked with Retry. No automatic retry was scheduled. **Steps to reproduce** 1. Use a Daytona sandbox environment with the default `timeoutMs` (300,000). 2. Start a run while Daytona keeps the new sandbox in `creating`. 3. See that the run waits 300 s, fails `setup_failed`, and is not retried. **Expected behavior** A create that has not started before provider work must not cost 5 minutes. The run must retry by itself within bounds. A late sandbox that has started must be used. A sandbox that was abandoned must be removed. **Root cause** - `packages/plugins/sandbox-providers/daytona/src/plugin.ts` (`onEnvironmentAcquireLease`): the only bound on the create step was the whole acquisition budget (`timeoutMs`, default 300 s). - `server/src/services/environment-provider-unavailable-error.ts` (`readTransientEnvironmentProviderFailure`): only an HTTP status or errno on the create call counted as transient. The timeout envelope carries neither, so the run fell through to generic `setup_failed`, which `isTransientWorkspacePreparationCode` does not retry. ## What Changed - Daytona: one fresh acquisition is now capped at 180 s. A create that has not returned after 90 s is looked up once by its creation name. A sandbox in `started` is adopted, and setup continues. A sandbox in `pending_build`, `building_snapshot`, or `pulling_snapshot` keeps waiting inside the cap. Any other state ends the attempt with the existing cleanup envelope and an acquisition diagnostic whose elapsed time reached the expired budget. - Daytona: a late SDK failure no longer runs the compensating delete for a sandbox that was adopted. - Host: `readTransientEnvironmentProviderFailure` reads the validated acquisition diagnostic. An acquisition whose first failure was its own deadline is transient. The run fails with `environment_provider_unavailable`, category `timeout`, and uses the bounded bootstrap retry (30 s, then 60 s). An earlier setup failure stays `setup_failed`. Message text is never parsed. - Host: the task reason for this case is plain: "The sandbox for ... did not finish starting in time. No agent work had started. Paperclip retries automatically ...". - Docs: `doc/execution-semantics.md` and `doc/plugins/SANDBOX_PROVIDER_CAPABILITIES.md`. ## Verification Red before the fix, green after: - `server/src/services/heartbeat/run-completion.test.ts` "routes a fresh sandbox create that timed out through bounded bootstrap recovery" uses the staging error shape through the orchestrator. Red: `errorCode` was `setup_failed`. Green: `environment_provider_unavailable`, category `timeout`, bounded retry scheduled. The `started` and `early_failure` cases stay `setup_failed` with no retry. - `server/src/services/environment-provider-unavailable-error.test.ts`: deadline diagnostic is transient (red, then green). An early failure is not. The task message is plain. - `packages/plugins/sandbox-providers/daytona/src/plugin.test.ts`: "ends a create still pending at 90 seconds" (red: still pending at 90 s; green) and "adopts a create that started by 90 seconds" (red; green, and a late SDK rejection does not delete it). "keeps waiting for a create that is still preparing its snapshot" is a safety case. - Existing tests that encoded the 300 s acquisition budget now use the 180 s cap. Commands run locally: - `npx vitest run src/plugin.test.ts` (Daytona): 414 passed. - `npx vitest run src/services/heartbeat/run-completion.test.ts src/services/environment-provider-unavailable-error.test.ts src/__tests__/environment-acquisition-diagnostics.test.ts src/__tests__/task-interruption-query.test.ts`: 229 tests passed. The run-completion file reports an `afterAll` embedded-Postgres teardown timeout. The same timeout occurs on unchanged `origin/master`, so it is a local platform fault. - Daytona plugin `pnpm typecheck` and server `pnpm typecheck`: pass. - `pnpm verify`: asked for broader CI coverage, because `doc/execution-semantics.md` is a global CI input. Required CI supplies that coverage. Upgrade safety: nothing new is persisted. A task already blocked by this failure keeps `canRetry: true`. Its Retry starts a new run with a new creation record, and that run uses the new path. On staging, the manual Retry of the affected task got a fresh lease in 28 s. The creation record of the earlier attempt stays with the creation reconciler. Not verified: a live Daytona create that stays in `creating`. That cannot be forced on demand. ## Risks - A legitimately slow create that stays in `creating` or `starting` for more than 90 s now ends the attempt and retries. Snapshot preparation states still wait inside the 180 s cap. Staging creates normally take about 20 to 30 s. - An abandoned create is removed by the creation reconciler. It can stay alive until that reconciler settles the record (the existing settle window). This PR does not change that window. - Other sandbox providers get the host classification only when they send the same acquisition diagnostic. ## Model Used Claude Opus 5.5 (claude-opus-5-5), Anthropic, used through Claude Code with tool use (shell, file edits, browser for read-only staging API reads). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude
|
889b33da72 |
fix(native): apply a saved accepted result on Retry instead of a new turn (#20819)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native run on a remote sandbox can finish with an accepted result, and then fail to save its workspace. Saved-work recovery saves the files again from the retained sandbox, with no provider turn. > - #20758 made a save that finds the sandbox disk full stop at once as `sandbox_disk_full` and hand the task to the board with a Retry. > - When the user frees space and presses Retry, the save now succeeds. But the Retry then started a new agent turn (`strategy: continue`). That turn repeats work that already finished and spends provider budget again. The accepted result was not applied (`replacementDenied: explicit_user_continuation`). > - The decision for this case: when a durable record proves that the run's result was already accepted, Retry applies that saved result and starts no new turn. > - This pull request makes Retry admission wait for the run's saved-work holds and then applies the accepted result through the existing preserved-result finalization. > - The benefit is that a Retry after a failed save finishes the task from the work that was already done, with no repeated provider work or spend. ## Linked Issues or Issue Description Refs #20758 (full-disk save rests as `sandbox_disk_full` with a Retry), Refs #20810 (chaos fault DF1 for a full disk on the final workspace save) **What happened?** A native run had an accepted, succeeded result. Its workspace save failed because the sandbox disk was full, and the task rested at `sandbox_disk_full` with a Retry. The user freed space and pressed Retry. The save succeeded, and then Retry admission started a new run with `strategy: continue`. The accepted result stayed unapplied (the finalization coordinator stayed in `terminal_failure`). Two paths caused this: 1. Retry admission let the Retry through as soon as the saved files were verified. That can happen before preserved-result finalization runs: the recovery worker dispatches the Retry right after one hold settles, while the hold that Retry added for the same run is still scheduled. On master this is deterministic in that order, and a race otherwise. 2. Preserved-result finalization reads the newest saved-work hold of the run. While that hold was still scheduled, it declined, and nothing held the Retry back. **Expected behavior** After the save succeeds, and a durable record proves that the run's result was already accepted, Retry applies that result. The run and task finish from it. No new `turn.start` and no new provider spend. If the save fails again with a full disk, the task keeps the `sandbox_disk_full` rest and a working Retry. If there is no proven accepted result, Retry keeps the previous behaviour and starts one run. **Steps to reproduce** 1. Give a native run an accepted, succeeded result whose copy-back failed (`native_workspace_sync_out_retry_exhausted`). 2. Make the saved-work recovery save fail with `remote_workspace_sandbox_disk_full`. The hold goes to the board. 3. Free space. Press Retry. Let recovery save the files, then dispatch the Retry before preserved-result finalization runs. 4. On master, a new run starts with `previousRunId` set to the source run, and the accepted result is not applied. ## What Changed - `preserved-result-finalization.ts`: the eligibility checks move into one read (`readPreservedResult`). They are unchanged: an accepted, succeeded result of this run, a completed saved-work recovery with a verified receipt, no Stop, no newer run, no current execution, and no applied generation. A new read-only `canResumePreservedResultFinalization` uses the same read without locks. `resumePreservedResultFinalization` keeps the finalizer's lock order (coordinator first) and takes the issue and run rows with NOWAIT, so it cannot deadlock with the finalizer, owner-stop confirmation or Retry dispatch; a busy row is retried briefly, and the last attempt waits for its locks. It also writes an activity entry, `issue.saved_result_finalization_resumed`, when it applies a result. - `explicit-native-continuation.ts`: Retry admission holds an accepted Retry (`evaluate`) of a run whose saved files are verified and whose accepted result is still held, with the wait `finalizing_saved_result`. It holds while a saved-work hold of that run is still `scheduled`, `running` or `awaiting_evidence` and the result is otherwise applicable (no Stop, newer run, other owner or current execution), or once `canResumePreservedResultFinalization` is true. In every other case admission is unchanged. - `heartbeat/run-control.ts` (`resumeTaskRetry`): after dispatch, if admission held the Retry with `finalizing_saved_result`, it calls `resumePreservedResultFinalization`. That coalesces the Retry into the source run with `strategy: finalize_result` and re-arms finalization. This is one key read after admission; the resumption runs only for a held Retry. The recovery worker already calls the same function after each saved hold, so either path finishes the work. - Tests: `workspace-recovery-disk-full.test.ts` now proves the new Retry behaviour (red on master), a dispatch that lands while another hold of the run is still saving (red on master), a Retry while the disk is still full, and three negative cases that keep one new run. `workspace-recovery-preparation-gates.test.ts` (s12 and RC-50b) and `explicit-native-continuation.test.ts` (copy-back + `task_retry`) encoded the old "new run" result for the same situation. They now expect the saved result to be applied. A fixture helper `recoverRunHolds` runs the recovery steps for every hold of a run, as the worker does. - `ui/src/components/WorkspaceExportRecovery.tsx`: at the `sandbox_disk_full` rest, the retained-file panel no longer shows its own "Recover saved files" button and no longer hides the task's Retry. Before, the API said `canRetry: true` and the note said "Retry saves the files again", but the board showed only "Recover saved files" (seen in the DF1 chaos wave). The panel now says to free space and use Retry. A new UI test covers it. `pnpm check:token-gates` is clean. - `doc/execution-semantics.md`: documents the Retry behaviour and its conditions. ## Verification Red/green (`cd server && npx vitest run src/services/workspace-recovery-disk-full.test.ts`): - On master source with the new tests: 3 failed, 3 passed (the 3 negative cases pass on master, as they should). Each failure shows a second heartbeat run (the new agent turn): `expected [ …(2) ] to deeply equal [ Array(1) ]`. - On this branch: 7 passed. A fourth negative case (an operator Stop while a hold is still settling keeps one new run) was added after review; it also passes on master. - Guard removal 1: with the `canResumePreservedResultFinalization` check replaced by `true`, the negative test "an operator Stop recorded on the run" fails (`expected [] to have a length of 1 but got +0`: the Retry waits forever instead of starting its one run). - Guard removal 2: with the "hold still saving" check removed, the test "holds a Retry dispatched while another saved-work hold of the run is still settling" fails (a second run starts). Related suites pass on this branch: `workspace-recovery-disk-full`, `workspace-recovery-preparation-gates`, `workspace-recovery-admission`, `native-exhausted-finalization-retry`, `explicit-native-continuation` (641 tests). On the first revision, `task-retry`, `workspace-recovery-retry-copyback`, `workspace-recovery-warm-handoff`, `preserved-native-response-wait.integration`, `deferred-execution-wait` and `deferred-execution-receipt-work` passed (317 tests). Other retry and recovery suites were not run locally; required CI `general_tests` covers them. After the review fixes, the five related suites above passed again (642 tests). `npx tsc --noEmit -p server/tsconfig.json` is clean. `pnpm verify --explain` selects a broad plan (exit 2) because `doc/execution-semantics.md` is a global CI input. Required CI supplies that coverage. ## Risks - Behaviour change: an accepted Retry of a terminal run with an accepted, succeeded result and verified saved files now applies that result and starts no run. Before, it started one new run. This is the intended change. It also covers the s12 and RC-50b recovery-handoff cases, which used to start a new run. - A held Retry waits while any saved-work hold of the run is still saving. The recovery worker drives those holds to `complete` or `exhausted`. When a hold ends `exhausted` or the records do not prove the result, the wait ends and the Retry starts one run, as before. A hold that stays in `awaiting_evidence` keeps the Retry waiting with "Paperclip is finishing the saved result." - Upgrade-safe: no persisted shape changes. The checks read existing records (`native_run_results`, `native_run_finalizations`, the recovery action evidence, workspace receipts). A record written by an older server that lacks any required fact keeps the old behaviour. An older server that reads the new state sees a coalesced Retry and a re-armed finalization, which it already handles. - No new work before `turn.start` on the normal path. Admission adds the eligibility read only when it has already proven an accepted result and verified saved files for a Retry of that run. `resumeTaskRetry` adds one key read after admission. - Lock order: `resumePreservedResultFinalization` keeps the finalizer's order (coordinator, then issue, then run). Because it can now run beside Retry dispatch (which locks the issue first), it takes the issue and run rows with NOWAIT. A busy row ends the attempt, and it retries up to 9 times at 100 ms. The 10th attempt waits for its locks, as this resumption always did, so a recovered result is never left unapplied. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file editing, test execution) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Nicky LeachandClaude Opus 5.5
|
369e238295 |
fix(projects): make workspace storage the project's only placement choice (#20835)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A repository project gets its working files from core staging or from a workspace storage plugin > - #20459 reported that an installed storage plugin attached every new repository project, so sandbox agents could not use those projects > - #20587 fixed creation with an **Available in** choice that selects storage > - #20651 then saved an environment on every new project and used it before the agent default to place runs. That solved a problem that did not exist: the project needs storage, not a run environment > - Creation saves the first environment of the chosen group, so a project silently sends every agent's runs to that environment. Project Configuration also shows two controls (Environment and Workspace storage) for one decision > - This pull request removes the project run environment and makes storage the only placement choice a project owns. Storage is fixed at creation > - The benefit is that runs follow the agent again, and project settings show one clear, read-only fact about storage ## Linked Issues or Issue Description Refs #20459. Follow-up to #20587 and #20651. Refs #11644. That issue asks for a project-level run environment override. This pull request removes the override that #20651 added for it, because the override is not necessary to fix #20459. #11644 stays a separate feature request. Related: #18176 (open) repairs placement through the legacy workspace-policy field. This pull request does not restore project placement, so it conflicts with the direction of #18176. ## What Changed - **Remove the project run environment.** Migration `0466` drops `projects.environment_id`. Run placement no longer reads the project. Runs use the agent default, then the instance default, then local execution. The low-trust task sandbox and the enforced sandbox policies keep their precedence. - **Keep the #20651 recovery and binding guards.** A recovered run still uses its admitted environment lease (source `run`). Company-binding checks before admission and during lease creation stay. - **Keep the creation choice.** **Available in** in New Project still selects storage. It still shows only when the environments need more than one storage owner. That can only occur when a storage plugin is installed. `POST /projects` with `environmentId` selects the storage of that environment and does not save the environment. - **Fix storage at creation.** `PATCH /projects/:id` rejects any storage change with `409 project_storage_locked`, from core to a plugin and from a plugin to core. Before, a core project could move to a plugin. - **Drop legacy writes.** Older clients can still send `executionWorkspacePolicy.environmentId`. The service removes it on create and update. - **Simplify Project Configuration.** Remove the Environment and Workspace storage selectors. Show an advisory **Available in** list of the environments that can use the project's storage. Show it when a storage plugin is installed or when the project uses one. When the project's plugin is not installed, show **Storage plugin unavailable**. Without a storage plugin, show nothing. - **Issue workspace card.** Remove the "project default" environment label. - **Shared contract.** Remove `Project.environmentId`, the `environmentId` project field, and `expectedStorageProviderPluginId` from the update schema. The create schema keeps `environmentId` and `expectedStorageProviderPluginId` from #20587. - **Docs and maps.** Update `doc/project-environment-compatibility.md`, `docs/api/goals-and-projects.md`, and the project feature and agent-access maps. - **SDK source pins.** Refresh `tests/runner-e2e/source-sdk-public-inputs.json` for the two shared files. The shared file blobs return to their values before #20651. The refresh also corrects the `pnpm-lock.yaml` pin and `lockSha256`: `master` pins an older lockfile blob than the one it contains. ## Verification - Targeted server suites pass (227 tests): `project-environments`, `project-workspace-provider.integration`, `heartbeat/environment-selection`, `execution-workspace-policy`, `environment-selection-route-guards`, `environment-run-orchestrator`, `openapi-routes`. - Related server suites pass (349 tests): `environment-service`, `environment-routes`, `plugin-workspaces.integration`, `issues-service`. - Targeted UI suites pass: `ProjectProperties.concurrency`, `IssueWorkspaceCard`, `NewProjectDialog`, `NewProjectDialog.managed-sandbox`, `ProjectDetail`. - New tests: - Creation saves storage, not an environment. Core and plugin projects both reject a storage change with `project_storage_locked`. - A legacy policy `environmentId` is dropped on create and update. - A new run follows the agent default for any project. A recovered run keeps its admitted lease after the agent default changes. - Project Configuration has no storage or environment selector. The note is hidden without a plugin. It lists the core or plugin environments for the project's storage. It shows **Storage plugin unavailable** for a removed plugin. - `pnpm --filter @paperclipai/server typecheck`, `pnpm --filter @paperclipai/ui typecheck`, and `pnpm --filter @paperclipai/shared typecheck` pass. - `pnpm --filter @paperclipai/db check:migrations`, `pnpm check:token-gates`, `node scripts/check-maps.mjs`, and `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check` pass. - Not run locally: the full `pnpm test:run`, `pnpm build`, and browser suites. CI supplies that coverage. - Manual check: open Project Configuration on an instance with a storage plugin. Confirm that there is no Environment or Workspace storage selector, and that **Available in** lists the environments. ## Risks - **Migration.** `0466` drops `projects.environment_id` and its foreign key. The migration deletes the saved project environments. Apply it before you serve the new revision. An old server revision that reads `projects` fails until it stops. This is the same pattern as earlier drop-column migrations (`0229`, `0289`). - **Behavior change for existing projects.** Projects created after #20651 shipped have a saved environment that overrides the agent. After this change, their runs follow the agent default again. This is the intended fix. Before #20651, a project environment selection was not applied to runs (#11644), so no older behavior is lost. - **No path from core to a plugin.** A core project, including one created before a plugin was installed, cannot move to the plugin. To use plugin storage, create a new project. - **API.** A `PATCH` that sends `environmentId` or `expectedStorageProviderPluginId` no longer has an effect. The schema strips these fields. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) by Anthropic, in Claude Code, with tool use (shell, file edits, test runs). Extended thinking was available. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Fable 5.2
|
39442400b7 |
fix(native): let a saved governed wait settle on its own same-run resume (#20838)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs can save a governed wait (a question or confirmation for a person) and park the provider turn > - If shutdown of the retained execution cannot be confirmed in time, the run keeps its checkpoint, records `native_governed_wait_settlement_pending`, and schedules a same-run resume that settles the saved wait (#20336, #20494, #20519) > - That failure writes an active recovery action for the run. Its cause is in `EXECUTION_RECONCILIATION_CAUSES`, and its wake policy resumes that same run > - When the run is a resolved-interaction continuation, its dispatch gate checks execution blockers. The gate read the run's own resume action as a hold and cancelled the resume with `execution_reconciliation_required` > - The task then stayed blocked with no Retry and no continue control, and the saved question could not wake the agent > - This pull request makes the gate skip only the run's own scheduled-resume action > - The benefit is that a saved governed wait settles by itself, with no person involved ## Linked Issues or Issue Description Refs #20494, #20519, #20809 **What happened?** A staging Codex run on a remote sandbox was a continuation after an answered question. The agent saved a second governed wait (a confirmation). Shutdown confirmation of the retained execution was still pending, so the run recorded `native_governed_wait_settlement_pending` and scheduled its exact-checkpoint resume. The resume reattached to the remote runner and started. 32 s later it ended `cancelled` with `execution_reconciliation_required`. No person cancelled it. The lifecycle event of the cancel carried the run's own recovery action id. The task showed `blocked` with `canRetry: false` and `canContinue: false`, and no later run started. **Expected behavior** The same-run resume settles the saved governed wait. The task waits for the person's answer. No run is cancelled and no recovery action needs a person. **Steps to reproduce** 1. A native run starts as a resolved-interaction continuation (context `mutation: interaction`, `interactionStatus: answered`). 2. The run saves a governed wait, and shutdown confirmation does not arrive in time (`native_governed_wait_settlement_pending`). 3. The coordinator schedules the same-run resume. At dispatch, `decideCurrentRunStaleness` calls `getExecutionBlocker`, which returns the run's own active action, and `cancelStaleRunInTx` cancels the run. ## What Changed - `server/src/services/execution-blocker.ts`: `getExecutionBlocker` takes `resumingNativeRunId`. For that run it skips only the active action whose wake policy is `resume_native_run` for the same run and whose evidence names that run. Workspace-restore holds and all other actions still block. - `server/src/modules/run-dispatch/adapters/postgres.ts`: the dispatch staleness decision passes the native run id. - `server/src/modules/run-dispatch/adapters/postgres.test.ts`: the failing case and two controls (an action for another run, an action without a resume policy still cancel). - `tests/restart-chaos`: new opt-in workload Q4 (a second saved question asked by the first answer's continuation run). Provider-fault rules take `occurrence`; Q4 rules use 2, so PV1/PV2 hit the second governed wait. Codex shim and ACPX tap both honor it. README documents Q4. - `doc/execution-semantics.md`: states that the run's own scheduled resume is not a hold against it. ## Verification - Red on master: `postgres.test.ts -t "saved governed wait"` fails with `{ dispatched: false, status: "cancelled", errorCode: "execution_reconciliation_required" }` (the staging error code). - Green: the same test passes; the two controls still cancel. - Focused: `postgres.test.ts`, `explicit-native-continuation.test.ts`, `native-preclaim-continuation.test.ts`, `workspace-recovery-preparation-gates.test.ts`; server typecheck. - Harness: `question-handoff.test.ts`, `provider-faults.test.ts`, `acpx-provider-faults.test.ts`, `journeys.test.ts`; harness typecheck. - Chaos: `--adapters codex --workloads Q4 --faults PV2 --tasks 1 --seed 111` on master and on this branch (results in a comment below). ## Risks - Low. The exemption is exact: same run id in the wake policy and the evidence, status `active`, only for a native run being dispatched. A recovery action that names another run, has no resume policy, or is resolved keeps its current effect. - Saved state from older versions: actions already written have the same shape, so a resume that is still scheduled settles after deploy. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude (Anthropic), model ID `claude-opus-5-5`, used through Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
DottaandClaude Fable 5.2
|
1d52436bf8 |
fix(config): detect the tailnet bind host only when the bind reads it, one time per process (#20843)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The server reads its configuration through `loadConfig()` in `server/src/config.ts`. Runtime code calls it often: `getStorageService()` calls it on each use, and the native runtime calls it for backup storage, response delivery and attachment reuse > - `loadConfig()` called `detectTailnetBindHost()` on each call. That function runs `tailscale ip -4` with `execFileSync`, which is a synchronous child process. The event loop stops until the child process ends > - A CPU profile of a busy hosted server showed this cost on the main thread, on a server that does not bind to a tailnet > - The tailnet address is an input only to the bind host, and only the listener that the server binds at startup uses the bind host. Most bind configurations never read the address > - This pull request makes `loadConfig()` detect the address only when the bind configuration reads it, and keeps the result for the life of the process > - The benefit is that a `loadConfig()` call no longer starts a child process on runtime paths, so the event loop stays free for requests ## Linked Issues or Issue Description No public issue exists. I searched open and closed pull requests for `tailscale loadConfig` and `detectTailnetBindHost` and found no related work. **What happened?** Each `loadConfig()` call started `tailscale ip -4` as a synchronous child process. A 30 second CPU profile of the main thread of a hosted server (commit |
||
|
DottaandClaude
|
3914d0c94a |
test(chaos): add DF1, a sandbox disk-full fault on the final workspace save (#20810)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart chaos harness (`tests/restart-chaos/`) runs real agent
tasks and breaks the control plane on purpose, so failures show up
before staging users see them
> - A census of 48 hours of staging failures found runs whose result
save-back failed because the sandbox disk was full (class F). The
harness never filled a sandbox disk, so it missed that class
> - #20758 made a full-disk save end recovery at once with cause
`sandbox_disk_full` and a Retry. Nothing in chaos exercised that path
end to end
> - A real full disk cannot be made on demand. The owner chose a
test-only hook that makes the save report a full disk, not a small disk
> - This pull request adds the hook at the save's sandbox boundary and a
DF1 fault that arms it, clears it, and grades the result
> - The benefit is that every wave can now check the disk-full rest, the
Retry offer, and that Retry works after space is freed
## Linked Issues or Issue Description
Refs #20758 (product behaviour under test), #20678 (finalization hands a
full-disk copy-back to recovery)
**What is the problem?**
The chaos harness had no fault for a full sandbox disk during result
save-back. Staging saw this failure (census class F in
`doc/plans/2026-11-11-staging-failures-vs-chaos-coverage.md`), and chaos
could not reproduce it.
**What is the expected behaviour?**
A wave can schedule DF1. While DF1 is armed, the task's final workspace
saves fail as a full disk. When DF1 is cleared, saves succeed again. The
task must rest with `sandbox_disk_full` and a Retry, and that Retry must
finish the task.
**Why does it matter?**
Without a fault, a regression in the disk-full path (no Retry, a wrong
cause, or a Retry that does not work) reaches staging users before chaos
sees it.
## What Changed
- `server/src/services/sandbox-disk-full-fault.ts` (new, test-only):
`sandboxDiskFullFaultTarget` wraps the target of a final save only when
the harness armed that save. The capture commands then exit 1 with the
exact message the in-sandbox script writes on ENOSPC, and the provider
download fails with `No space left on device`. The gate follows
`@paperclipai/shared/crash-points`: `PAPERCLIP_DISK_FULL_FAULT_CONTROL`
is read once at import and ignored under `NODE_ENV=production`.
Disarmed, the hook is one comparison and returns the save's own target.
- `server/src/services/remote-workspaces.ts`: the capture call in
`saveRemoteWorkspace` passes its target through the hook. No other
product change. `workspace-recovery.ts` is not touched.
- `tests/restart-chaos/disk-full-faults.ts` (new): the DF1 kind, the
control-file seam (atomic writes), the rest evidence
(`activeRecoveryAction.evidence.systemRecovery.failure.cause.label`),
the `disk_full_fault` T check, and the DF1 R3 rule (one expected Retry
that must finish the task).
- Harness wiring: `--faults DF1` (refused without `--env daytona`, with
a storm, or with PV faults), no scheduled restart, arm at launch, clear
at the first rest just before the Retry click, receipts in
`<wave>/disk-full-faults.jsonl`, and ledger fields `diskFull: {
faultedSaves, restedDiskFull, retryOffered, retryWorked }`.
- Docs: a "Sandbox disk full (DF1)" section in
`tests/restart-chaos/README.md`, and the census plan marks DF1 as built.
## Verification
- `cd server && pnpm exec vitest run
src/services/sandbox-disk-full-fault.test.ts`: the gate is off by
default and under `NODE_ENV=production`, the save's own target comes
back when disarmed, only a listed final save is faulted, the product
capture turns it into the named disk-full failure, and clearing the file
disarms the next save.
- `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts
tests/restart-chaos/disk-full-faults.test.ts
tests/restart-chaos/grader.test.ts tests/restart-chaos/scheduler.test.ts
tests/restart-chaos/provider-faults.test.ts`: planning, the control
file, and grading.
- `node scripts/typecheck-test-projects.mjs tests/restart-chaos` and the
server `tsc --noEmit`: clean.
- Live Daytona waves (Codex, `--storage s3`, image content `96c4339f…`,
run through the two-slot queue):
- `w-20261111T131314-s9411` (B1): the hook was armed in the server but
no save reached it. B1 uses a local worktree synced into the sandbox,
not a remote workspace. Graded `not_applied` as designed. The harness
now requires `--storage s3` and the docs point at R1.
- `w-20261111T131941-s9412` (R1): 3 faulted saves. t01 rested `blocked`
with `sandbox_disk_full`, `user_action_required`, board owner, API
`canRetry: true`. t02 was cleared too early, while recovery still
checked its source; that recovery then saved and the task finished.
Fixed in
|
||
|
DottaandClaude Opus 5.5
|
34f2bb78ee |
feat(connections): connect Sentry with a long-lived user auth token (#20823)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents use app connections to call provider tools, such as the
Sentry MCP server
> - The Sentry connection has only one method: browser sign-in through
the hosted MCP OAuth server
> - That grant expires. An unattended triage agent loses Sentry access
until a person signs in again
> - During that time, a scheduled triage routine completes each run but
reads no Sentry data
> - This pull request adds a second Sentry method that uses a user auth
token
> - The benefit is a Sentry connection that stays valid until a person
revokes the token
## Linked Issues or Issue Description
**Subsystem affected**
App connections: the Sentry entry in the connection catalog.
**Problem or motivation**
A Sentry connection that was created with browser sign-in on 2026-10-11
changed to "OAuth authorization expired. Reconnect this app to
continue." at 04:01 UTC on 2026-11-10. The agent that used it had no
Sentry tools after that time. Each hourly triage run reported the
blocker but could not continue. Unattended agents need a Sentry
connection that does not need a person to sign in again each month.
**Proposed solution**
The hosted Sentry MCP server accepts a Sentry user auth token in the
`Authorization: Sentry-Bearer <token>` header. It reserves `Bearer` for
its own OAuth tokens. Paperclip API-key methods already support a custom
header prefix. Add a Sentry API-key method that uses this header.
**Alternatives considered**
Reconnect with browser sign-in again. That restores access only until
the next grant expiry.
## What Changed
- Add the `mcp-user-token` method ("Use a user auth token") to Sentry in
`scripts/ingest-app-definitions.mjs`. It uses the same server URL and
sends the token as `Authorization: Sentry-Bearer <token>`.
- Regenerate `packages/shared/src/app-definitions/sentry.json`. I did
not include unrelated output that the content corpus drift caused for
other providers.
- Add a permission review entry for the new method in
`doc/connections/tool-method-permission-reviews.json`.
- Add a test for the new method and its header prefix.
- Document the token method, its required scopes, and its ownership risk
in `doc/connections/SENTRY.md`.
## Verification
- `pnpm exec vitest run packages/shared/src/app-definitions.test.ts`: 40
passed. The new assertion fails on `master` because Sentry has only one
method there.
- `pnpm exec vitest run src/__tests__/tool-access-service.test.ts -t
Sentry` (server): 2 passed.
- `pnpm exec vitest run ui/src/pages/apps/app-connect-policy.test.ts
server/src/__tests__/satisfied-connection-intents.test.ts`: 64 passed.
- `pnpm verify` returned exit 2 (a doc is a global CI input). Required
CI supplies the broad coverage.
- Live probe of `https://mcp.sentry.dev/mcp` with an `initialize`
request: `Authorization: Sentry-Bearer <invalid token>` starts an MCP
session (the token is checked on tool calls). `Authorization: Bearer
<same token>` returns `401 invalid_token`. This confirms the server
accepts the header scheme.
- Not done yet: a real agent tool call with a real token after
deployment.
## Risks
- Low risk. This adds a method. The existing OAuth method and saved
connections do not change.
- A user auth token acts as its creator and lasts until revoked. The
method guidance and docs say this. Rotate the token when that person
leaves.
- If Sentry changes the `Sentry-Bearer` scheme, connections that use
this method fail on tool calls. The OAuth method still works.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, browser, web search).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandClaude Opus 5.5
|
e2cf9a670f |
fix(runner-e2e): resync SDK public lock pins and make the lockfile bot keep them current (#20623)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Runner support suite proves the plugin SDK public types from pinned sources in `tests/runner-e2e/source-sdk-public-inputs.json` > - That registry pins the committed `pnpm-lock.yaml` blob and a `lockSha256` that must equal the `outputSha256` in `tests/runner-e2e/report-lock.json` > - The lockfile bot refresh #20615 changed the lock and `report-lock.json`, but the bot does not write the SDK registry > - The bot merges without CI, so the drift guard from #20579 could not block it, and every PR now fails `ci / Runner support suite` > - The guard also checked only the source blobs, so it did not name the `lockSha256` drift, and the sync script could not update `lockSha256` > - This pull request regenerates the registry, derives `lockSha256` from the reviewed policy, and makes the bot resync and check the registry in its own commit > - The benefit is that master CI is green again, and the next lockfile refresh cannot leave the SDK proof stale ## Linked Issues or Issue Description Refs #20615. Refs #20579. Refs #20550. Refs #20613. Overlaps #20618, which refreshes only the registry data. This PR has the same data change and also closes the gap that let the drift land. **What happened?** After #20615 merged, `ci / Runner support suite` failed on master and on every PR (for example runs 42877750315 and 42877750957, and PR #20613). `source-sdk-public-type.test.ts` and `source-sdk-evidence.test.ts` failed with "Reviewed SDK reference policy does not match public proof". `source-sdk-public-inputs.test.mjs` failed with "pnpm-lock.yaml (changed)". **Expected behavior** A lockfile refresh keeps the SDK public proof registry in step with the lock and the reviewed report-lock policy. If the registry drifts, the failure names the stale pin and the fix command. **Steps to reproduce** 1. Check out master at `3bf7af6c77`. 2. Run `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts source-sdk`. 3. See the "Reviewed SDK reference policy does not match public proof" failure. ## What Changed - Root cause: #20615 changed `report-lock.json` from output `2360d5e9…` to `b3208306…` and changed the `pnpm-lock.yaml` blob from `851e65af…` to `6d825f97…`. The registry kept the old values. `refresh-lockfile.yml` commits only the lockfiles and the report-lock proof, and it merges without CI. - `source-sdk-public-inputs.json`: regenerated with `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --write`. Only `inputs["pnpm-lock.yaml"]` and `lockSha256` change. The compiler, parser and Zod pins do not change. - `source-sdk-public-inputs.mjs`: the sync now copies `lockSha256` from the committed `report-lock.json` `outputSha256`. It refuses when the committed lock is not that policy's reviewed input or output. This is the same rule that `prepareReviewedSdkReference` applies. Only the lockfile bot can write that policy, so a contributor cannot choose `lockSha256`. - `staleSdkPublicInputs()`: also reports a `lockSha256` that differs from the committed report-lock output. The guard test then names both stale pins and the `--write` command. - `source-sdk-reviewed-fixture.ts`: the "Reviewed SDK reference policy does not match public proof" error now names the two values and the fix command. The check itself is unchanged. - `refresh-lockfile.yml`: after the lock commit, the bot runs the SDK sync with `--write`, amends the commit, and runs `--check` before it pushes. `head_sha` is still read after the amend, so the bot merges the exact commit it checked. - Tests: new sync tests for a lockfile refresh and for an unreviewed lock, and a workflow-order test in `lockfile-refresh-merge.test.mjs`. - Docs: `doc/RELEASE-AUTOMATION-SETUP.md` §8.1 and `tests/runner-e2e/OPERATIONS.md` describe the resync. ## Verification - Reproduced on master: `source-sdk-public-type.test.ts` and `source-sdk-evidence.test.ts` failed with the policy mismatch. - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts source-sdk prepare-report-lock sdk-canary workflow-security`: 9 files, 135 tests pass. - Before the registry change, the extended guard reported `pnpm-lock.yaml (changed), tests/runner-e2e/report-lock.json outputSha256 (lockSha256 changed)` with the fix command. - `node tests/runner-e2e/source-sdk-public-inputs.mjs $(git rev-parse HEAD) --check` reports `changed: false`. - `node --test .github/scripts/tests/lockfile-refresh-merge.test.mjs .github/scripts/tests/lockfile-refresh-workflows.test.mjs .github/scripts/tests/lockfile-trust.test.mjs`: 39 tests pass. - `pnpm test:e2e:runner:unit` on a host with load near 28: 6 files timed out, and none of them are the SDK proof files. All 15 of those files pass when run alone. CI runs the full suite. ## Risks - The bot step now runs one more repository script. If that script refuses (for example, an unreviewed committed lock), the refresh PR is not pushed. Before this change, the bot merged a refresh that made master red. - `lockSha256` now comes from the reviewed policy, not from the old registry value. The proof check is unchanged. It still needs the lock bytes to hash to `lockSha256`. - If #20618 merges first, this branch rebases cleanly, because the registry change is identical. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edit) and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude
|
fd1460bb24 |
fix(native): continue a governed-wait follow-up in a fresh session when its saved provider state is gone (#20809)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness (`tests/restart-chaos/`) runs real agent tasks and breaks the control plane on purpose, so failures show up before staging users see them > - A census of 48 hours of staging failures found runs whose result save-back failed because the sandbox disk was full (class F). The harness never filled a sandbox disk, so it missed that class > - #20758 made a full-disk save end recovery at once with cause `sandbox_disk_full` and a Retry. Nothing in chaos exercised that path end to end > - A real full disk cannot be made on demand. The owner chose a test-only hook that makes the save report a full disk, not a small disk > - This pull request adds the hook at the save's sandbox boundary and a DF1 fault that arms it, clears it, and grades the result > - The benefit is that every wave can now check the disk-full rest, the Retry offer, and that Retry works after space is freed ## Linked Issues or Issue Description Refs #20758 (product behaviour under test), #20678 (finalization hands a full-disk copy-back to recovery) **What is the problem?** The chaos harness had no fault for a full sandbox disk during result save-back. Staging saw this failure (census class F in `doc/plans/2026-11-11-staging-failures-vs-chaos-coverage.md`), and chaos could not reproduce it. **What is the expected behaviour?** A wave can schedule DF1. While DF1 is armed, the task's final workspace saves fail as a full disk. When DF1 is cleared, saves succeed again. The task must rest with `sandbox_disk_full` and a Retry, and that Retry must finish the task. **Why does it matter?** Without a fault, a regression in the disk-full path (no Retry, a wrong cause, or a Retry that does not work) reaches staging users before chaos sees it. ## What Changed - `server/src/services/sandbox-disk-full-fault.ts` (new, test-only): `sandboxDiskFullFaultTarget` wraps the target of a final save only when the harness armed that save. The capture commands then exit 1 with the exact message the in-sandbox script writes on ENOSPC, and the provider download fails with `No space left on device`. The gate follows `@paperclipai/shared/crash-points`: `PAPERCLIP_DISK_FULL_FAULT_CONTROL` is read once at import and ignored under `NODE_ENV=production`. Disarmed, the hook is one comparison and returns the save's own target. - `server/src/services/remote-workspaces.ts`: the capture call in `saveRemoteWorkspace` passes its target through the hook. No other product change. `workspace-recovery.ts` is not touched. - `tests/restart-chaos/disk-full-faults.ts` (new): the DF1 kind, the control-file seam (atomic writes), the rest evidence (`activeRecoveryAction.evidence.systemRecovery.failure.cause.label`), the `disk_full_fault` T check, and the DF1 R3 rule (one expected Retry that must finish the task). - Harness wiring: `--faults DF1` (refused without `--env daytona`, with a storm, or with PV faults), no scheduled restart, arm at launch, clear at the first rest just before the Retry click, receipts in `<wave>/disk-full-faults.jsonl`, and ledger fields `diskFull: { faultedSaves, restedDiskFull, retryOffered, retryWorked }`. - Docs: a "Sandbox disk full (DF1)" section in `tests/restart-chaos/README.md`, and the census plan marks DF1 as built. ## Verification - `cd server && pnpm exec vitest run src/services/sandbox-disk-full-fault.test.ts`: the gate is off by default and under `NODE_ENV=production`, the save's own target comes back when disarmed, only a listed final save is faulted, the product capture turns it into the named disk-full failure, and clearing the file disarms the next save. - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts tests/restart-chaos/disk-full-faults.test.ts tests/restart-chaos/grader.test.ts tests/restart-chaos/scheduler.test.ts tests/restart-chaos/provider-faults.test.ts`: planning, the control file, and grading. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos` and the server `tsc --noEmit`: clean. - Live wave: a Daytona DF1 wave is running. Its result is added here before merge. ## Risks - Low risk for production. The hook reads its variable once at startup and is ignored under `NODE_ENV=production`. Without the variable it does one comparison and returns the same target object. - The hook fails every sandbox command of an armed capture, including the read-only fallback's inventory. A real full disk can let that inventory succeed and fail later at the download. The outcome is the same (`sandbox_disk_full`), but the fallback's inventory step is not exercised under DF1. - DF1 runs only with `--env daytona`. A local run has no sandbox workspace save, so the harness refuses DF1 there. ## Model Used - Claude Opus 5.5 (Anthropic, model ID `claude-opus-5-5`), in Claude Code with tool use (shell, file edits, GitHub CLI). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
5f38226290 |
fix(build): report compiler signals in server typecheck layers (#20603)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Its development checks help contributors verify changes.
> - The server typecheck runs the compiler once for each source layer.
> - A compiler child can stop on a signal before it reports type errors.
> - The wrapper returned failure without the signal or the affected
layer.
> - This pull request reports the signal, layer, command, and working
directory.
> - Contributors can identify an interrupted check from its output.
## Linked Issues or Issue Description
**What happened?**
The server typecheck returned exit 1 after a layer banner when a
compiler child stopped on a signal. The wrapper did not report that
signal. Related layer work: #18737, #18738, and #18739. A search of open
and closed PRs found no duplicate signal-reporting fix.
**Expected behavior**
Report the actual termination signal and affected layer. Keep compiler
output visible and return failure. Do not infer a memory failure from
SIGKILL alone.
**Steps to reproduce**
Run a controlled compiler child that sends SIGTERM or SIGKILL to itself.
Run it through typecheckLayers and the real runTsc subprocess path. The
added tests do this without memory pressure or timed waits.
**Paperclip version or commit**
Reproduced on
|
||
|
DottaandPaperclip
|
81c9143fe4 |
fix: upgrade Better Auth to 1.7.16 with account migration (#18906)
## Thinking Path > - Paperclip manages AI agents and their work. > - The board uses Better Auth for credential accounts and sessions. > - The server pins Better Auth 1.7.2. > - The November security release fixes 12 issues in Better Auth. > - The new version stops writing the old required account issuer field. > - This change updates the packages and the database contract together. > - New users can sign up, and existing users can still sign in. ## Linked Issues or Issue Description **What happened?** The server uses Better Auth 1.7.2. The [security release](https://better-auth.com/blog/security-release-2026-11-03) fixes 12 issues in 1.7.16. Earlier update PRs #17909 and #17911 target 1.7.7, which is below this security release. **Expected behavior** Use the patched stable release. Keep new and existing credential accounts usable. **Steps to reproduce** Read the server dependency pin. Upgrade it without a schema change. Better Auth then rejects the required account issuer column. See the [upgrade guide](https://better-auth.com/docs/guides/1-7-upgrade-guide). ## What Changed - Pin Better Auth to 1.7.16. The repository refresh bot owns the committed Core lockfile. - Make the old issuer column nullable. Keep existing issuer values. - Replace the issuer index with a unique provider/account index. - Generate the migration and retain the five newest schema snapshots. - Test new signup, new-account sign-in, legacy issuer sign-in, duplicate rejection, and separate provider identities with real PostgreSQL. - Update the database schema test. Test the historical issuer backfill separately. Verify upgrades over existing rows and rollback on duplicate provider/account keys. - Document the duplicate check and migration order. - Add the approved two-line dependency resolution step before the backup job’s frozen install. Leave the frozen check and backup tests intact. ## Verification - Core head: `af70282c7e24335b8688c8f3cc88fdf65045632c`; parent `0f1481e4462100357a7caa496b875cf0bc500d95` includes master `84fc67317`. - Nine changed files. The only workflow edit is the approved two-line backup dependency resolution step; no managed lockfile edits. - Migration `0465_better_auth_account_identity.sql` is three SQL statements; snapshot changes only `public.account`, five newest snapshots retained. - DB typecheck and 23 focused tests passed on the parent. Both real PostgreSQL migration/signup suites were rerun November 11: all 4 tests passed. Application source and tests are unchanged in this workflow-only commit. - The parent’s sandbox-provider lane passed all 205 file-sync tests but reported an unhandled EPIPE during the symlink rejection case; its aggregate verify therefore failed. No unrelated source change was made. The new head reruns the full applicable CI. - Reproduced the old backup install failure locally. Resolving manifests followed by a frozen install passed locally; restored the bot-owned committed lock afterward. Hosted backup qualification remains required. - Current-head CI and review are running; no final-head green claim or merge yet. ## Risks Apply the migration before the new server starts. Check for duplicate provider/account pairs first. Duplicate rows stop index creation. Keep separate users separate when resolving duplicates. The migration keeps issuer data. Do not restore its NOT NULL constraint while new rows contain null. No deployment is included. ## Model Used OpenAI GPT-6 Codex, with reasoning and tool use. The runtime did not expose a more specific model ID or context window. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
aaf1337c7b |
fix(slack): require mentions only for channel roots (#20821)
## Thinking Path > - Paperclip helps people manage AI agents for work. > - Slack connects people to those agents through channels and threads. > - A mention can start a conversation without making the bot answer all channel traffic. > - The current setting also requires mentions on every thread reply. > - This interrupts an existing conversation and can cancel a queued reply. > - This change requires mentions on channel roots and allows replies without a mention. > - People can continue a thread without adding the bot name to each message. ## Linked Issues or Issue Description **What existing behavior does this improve?** The Slack connection setting that requires an at-mention. **Current behavior** When the setting is on, a channel thread reply must mention the bot. The delivery and execution checks reject a reply that has no mention. **Proposed behavior** Require a mention on a top-level channel message. Allow replies in existing threads without a mention. Keep the current DM access controls. **Reason and benefit** A person can continue an existing conversation without repeating the bot name. **Breaking changes** Existing connections with the setting on will accept thread replies without a mention. The saved setting, default value, and API field stay the same. ## What Changed - Rename the control to **Require at-mention in channels**. Explain the thread and DM rules in the help text and operator documentation. - Keep the mention policy in the Slack module. Use the thread root and message IDs to identify a reply. - Call that policy from durable admission, wake authorization, and queued run start. These shared paths must enforce the same setting when a delivery resumes. Other providers keep their current behavior. - Update PostgreSQL integration tests for thread replies, DMs, queued deliveries, and combined wake inputs. Add tests that reject incomplete reply identity. ## Verification - Reproduced rejected thread replies and cancelled queued reply runs before the fix. - Passed 16 focused PostgreSQL integration cases after the fix. - Passed 71 tests on revision `31b1adede2b92bad24b8fa21b8f6fffa9bd26d04` after the rebase. This includes the full durable wakeup suite, Slack policy tests, and eight selected Slack delivery cases. - Command: `pnpm exec vitest run server/src/services/chat-channels/slack/wake-authority.test.ts server/src/__tests__/durable-chat-wakeup.test.ts server/src/__tests__/chat-channels.integration.test.ts -t "at-mentions by default|thread replies without mentions|rechecks queued Slack|preserves running Slack tool authority|Slack channel mention policy|durable inbound chat scheduler receipts"`. - Strengthened the queued-root test to check the mention-rejection reason and content redaction. A temporary change that disabled mention enforcement caused this test to fail with the unrelated-message reason. The real policy is restored. - Removed the added source comments to follow the contribution rules. - `node scripts/check-module-boundaries.mjs` and `git diff --check` pass. - `pnpm verify` returns exit 2 because the operator documentation is a global CI input. Required CI must supply the wider coverage. - Local preflight installed dependencies and built the plugin SDK. It could not build Runner Rust binaries because this environment has no `cargo` executable. The focused server tests run without those binaries. - Live Slack and browser tests were not run. This change uses the existing provider message and thread identities. - Required CI passed on `31b1adede2b92bad24b8fa21b8f6fffa9bd26d04`: [PR workflow](https://github.com/paperclipai/paperclip/actions/runs/43035551294), including typecheck, build, server/chat tests, Runner checks, and browser E2E. Current-head Greptile and Paperclip reviews are both 5/5 with no open findings. ## Risks - Existing connections with mention gating on will now accept unmentioned thread replies. This is the intended behavior change. - A missing or invalid thread or message ID does not grant the reply exception. - No schema migration is required. Destination, sender, and company access checks stay in place. - The existing chat integration test file is large. This change updates its current Slack cases to reuse the full delivery and authorization fixture. ## Model Used OpenAI Codex, based on GPT-6, with reasoning and code execution tools. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
fa3cd6f330 |
fix(native): end provider startup rejections with Retry instead of an ownership hold (#20808)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The Paperclip Runner starts Codex for native runs. On a remote
sandbox such as Daytona, the sandbox is the OS boundary.
> - A Codex planning run on Daytona failed in `session.open`. Codex's
read-only sandbox helper could not set up loopback in the container
(`bwrap: loopback: Failed RTM_NEWADDR`), and Codex loads `AGENTS.md`
through that helper.
> - The server treated that definite provider rejection as an
interrupted transport. Its same-run bootstrap retry went through remote
restart recovery and stopped on `ambiguous_remote_recovery_evidence`.
The task showed "Recovering saved work…" and no Retry, although no
provider work was done.
> - This pull request gives such startup rejections a named, permanent
outcome with a working Retry, and pins ask and standard runs to the
external sandbox policy.
> - The benefit is that a provider startup refusal ends as a clear
failure with Retry, never as an ownership hold.
## Linked Issues or Issue Description
**What happened?**
A Codex planning run on a Daytona sandbox failed in `session.open` with
`error creating thread: failed to load AGENTS.md instructions ... bwrap:
loopback: Failed RTM_NEWADDR: Operation not permitted`. The server
recorded `native_session_interrupted` with recovery mode
`bootstrap_retry`. Thirty seconds later, remote restart recovery blocked
the run with `native_restart_recovery_blocked` /
`ambiguous_remote_recovery_evidence`. No server restart happened. All
default-mode Codex runs on the same image succeeded that day; only the
planning run failed.
**Expected behavior**
A provider startup failure with no provider work must end as a clear
failure with a working Retry, or get a bounded automatic retry that also
ends with Retry. It must never become an ambiguous-ownership hold.
**Steps to reproduce**
1. `cd server && npx vitest run
src/services/native-runtime/native-startup-rejected-retry.integration.test.ts`.
On master the deterministic case is `retryable_failure` /
`native_session_interrupted`.
2. `npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t "provider
itself rejected"`. On master the new code is not produced.
**Paperclip version or commit**
master at
|
||
|
DottaandClaude Opus 5.5
|
4c04c20c9e |
fix(recovery): wake a parent whose refused completion waited on dependencies that later clear (#20803)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A parent task that waits on blocker tasks resumes through the `issue_blockers_resolved` wake > - A periodic dependency sweep sends that wake when a blocked task's dependencies become ready and no wake covers the current blocked cycle > - The sweep skips a task that its own agent blocked, because that agent waits on something outside the blocker graph > - A native completion that Paperclip refuses for unresolved dependencies is also recorded as a block by the agent's own run (`durable_dependency_blocker_bound`). The sweep treated it as a self-block. It woke the task only if a blocker later completed or finalized on its workspace > - When readiness cleared another way, nothing woke the task. It stayed `blocked` with no next action > - This pull request makes the sweep treat that refusal as a dependency wait, and count a "Recover saved files" finalize without a workspace as the blocker settling > - The benefit is that a blocked task is not a silent dead end: it resumes once, automatically, when its dependencies clear ## Linked Issues or Issue Description Refs #20728 **What happened?** A parent task had three done child tasks that block it. One child was still listed as "saving its files", because a "Recover saved files" finalize had no workspace. The parent's agent tried to finish. Paperclip refused the completion because dependencies were unresolved and set the parent `blocked` (`durable_dependency_blocker_bound`). After #20728 was deployed, the parent's dependencies were ready. But the task stayed `blocked` with no wake, no Retry, and no next action. A person had to post a comment to wake it. **Expected behavior** When the last blocker of a blocked task clears, by status or by finalize settlement, Paperclip wakes the assignee once. This also applies to tasks already blocked before a deploy that changes readiness. **Steps to reproduce** 1. Block a task on a done blocker whose workspace finalize is still pending. 2. Let the assignee agent's native run report the task done. Paperclip sets the task `blocked` with `durable_dependency_blocker_bound`. 3. Make the blocker settle without a later blocker completion or a finalize on its workspace (for example, a recovery finalize with no workspace, or a deploy that changes the readiness rule). 4. Run the dependency sweep. Before this fix it reports `selfBlockedSkipped: 1` and sends no wake. ## What Changed - `server/src/services/recovery/service.ts` (`wasBlockedByAssigneeAgent`): a native block with reason `durable_dependency_blocker_bound` is not a self-block. The dependency owns its wake. Other native blocks and agent blocks keep the self-block rule. - The same function counts a later succeeded `workspace_finalize` with `owningService: workspace_recovery` and no workspace as the blocker settling, which matches the readiness rule from #20728. - `server/src/__tests__/heartbeat-issue-liveness-escalation.test.ts`: regression tests for the refused-completion shape (one wake, no second wake), the recovery finalize without a workspace, and a safety case for a different native block reason. - `server/src/services/workspace-recovery-dependency-finalize.test.ts`: a characterization test that a real "Recover saved files" signals the dependency sweep for the company. This path already existed; the test keeps it. - `doc/execution-semantics.md`: documents when the dependency sweep wakes after the last blocker clears. ## Verification - Red (unfixed `service.ts`): `npx vitest run src/__tests__/heartbeat-issue-liveness-escalation.test.ts -t "refused for unresolved|still holds a native block|recovery finalize without a workspace"` fails 2 of 3 tests with `selfBlockedSkipped` 1 instead of 0. - Green (fixed): `src/__tests__/heartbeat-issue-liveness-escalation.test.ts` passes 44 of 44. - Related suites pass: `src/__tests__/issue-dependency-wakeups-routes.test.ts`, `src/__tests__/heartbeat-dependency-scheduling.test.ts`, `src/services/workspace-recovery-dependency-finalize.test.ts`, `src/__tests__/issue-recovery-dependencies.test.ts` (43 tests). Server `tsc --noEmit` passes. - `pnpm verify` exits 2 because a doc change is a global CI input. Required CI supplies broad coverage. - Staging evidence (read only): the affected parent's activity shows `native-status-committer` set it `blocked` with `durable_dependency_blocker_bound`, and no later blocker completion or finalize. ## Risks - Previous-release story: no persisted shape changes. Tasks already `blocked` with `durable_dependency_blocker_bound` and ready dependencies are picked up by the existing dependency sweep after deploy (it visits every blocked task at startup and on recovery signals). No migration. - Bounded: the wake uses the cycle-aware ready-state key, so one blocked cycle gets one wake. The sweep keeps its live-run, queued-wake, pending-interaction, and pause-hold checks. - No loop: if the woken agent blocks the task again while its dependencies are ready, the reason is not `durable_dependency_blocker_bound`, so the self-block rule applies. - The recovery finalize rule only applies to a finalize on the blocker task itself with `owningService: workspace_recovery` and no workspace. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, code search, test execution, read-only staging API inspection). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
556b249897 |
fix: preserve exact Markdown in native document writes (#20605)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agents save Markdown documents through native tools and REST routes. > - The native write_document handler trims the supplied body. > - This removes final newlines and can change Markdown whitespace. > - This pull request preserves the original body after required-content validation. > - Agents can now write and read the same Markdown string across these paths. ## Linked Issues or Issue Description **What happened?** The native write_document tool removes leading and trailing whitespace. A body with one final newline loses that newline on read-back. The REST document PUT and write_task_document preserve it. **Expected behavior** The native write_document tool must preserve the supplied Markdown body exactly. Required-content checks and key and title normalization must stay in place. REST-backed paths retain surrounding whitespace and keep their existing escaped-line-break conversion for single-line payloads. **Steps to reproduce** 1. Write a disposable document through write_document with a final newline. 2. Read the document and compare the full body with the supplied string. 3. Repeat through write_task_document and REST. These paths preserve the string. Related work: Refs #18824. That change aligns document keys. This change preserves body strings. A search of open and closed PRs found no duplicate fix. ## What Changed - Use a separate untrimmed required-string helper for the document body. Keep the existing requiredString callback signature and all other field normalization. - Add real tool-handler, database, and authenticated HTTP tests for create, update, read-back, and replay. Cover leading spaces, hard breaks, blank lines, final newlines, no final newline, Unicode, and mixed line endings. - Keep blank and invalid bodies rejected. Document whitespace preservation and the existing REST escaped-line-break exception. - Fix the Runner support archive-scanner shutdown defect exposed by CI: rejected ZIP scanners now receive SIGKILL so an ignored SIGTERM cannot leave publication waiting for process closure. Keep all secret and archive limits unchanged. A real child-process regression fails before the change and passes afterward. - Address all three initial review findings: callback type compatibility, contribution comment policy, and the REST documentation limit. ## Verification Final tested head: `c9ff8cf3f83aa19e5d8313c65816a3a2e6562368`. - All applicable [CI checks](https://github.com/paperclipai/paperclip/actions/runs/43019885807) passed. This includes typecheck, build, Runner support, Rust, native Runner, general tests, serialized suites, all eight E2E shards, and the required aggregate gates. - [Greptile review](https://github.com/paperclipai/paperclip/runs/130520170309) passed on this head with confidence 5/5. All three review threads are resolved. The independent Paperclip review also reports 5/5 with no findings. - Red at `7368a0b15c2f02107778a961144d86e2eb59a952`: `pnpm --dir server exec vitest run src/services/native-runtime/task-document-body.integration.test.ts` failed because write_document removed the final newline. Cross-task and REST cases passed. - Green on the final source: 176 tests passed across the document body, document key, native tool authority, task scope, document service, and document route suites. The tests cover create, update, exact read-back, replay, blank rejection, revision guards, and company scope. - The ZIP shutdown regression failed before repair with “scanner did not stop”. After repair, 54 tests passed across zip-inspection-shutdown, private-reporter-artifacts, manual-private-cell-report-assets, and private-evidence-budget. The regression uses a real process that ignores SIGTERM. - Earlier CI runs had two archive-related timeouts. The source repair above replaces blind reruns. The Runner support suite and aggregate verify gate now pass. - A disposable live native document reproduced the original whitespace loss. REST preserved the same string. The fixed handler passed local integration tests. The running service has not been deployed. - Local environment: Linux x64, Node 25.9.0, Vitest 5.0.3, and embedded PostgreSQL. Local Rust preparation was unavailable because cargo is absent. Final-head CI passed Rust verification. `pnpm verify --explain` requested broad CI because evidence.ts is a global input. Required CI supplied that coverage. ## Risks Low risk. The change retains required-content validation. The product change affects document body whitespace handling. The CI repair forcefully stops only already-rejected archive scanners; it retains fail-closed publication and all existing scan limits. It does not change schema, authorization, revision guards, receipts, or persistence. ## Model Used CI repair continuation: GPT-6.1, using repository tools and code execution. OpenAI Codex, GPT-6 (the exact serving model ID and context window were not exposed in this session). Used reasoning, repository tools, and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
84fc67317e |
feat(runner): export authorized read responses as artifacts (#20606)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native agents use authorized API reads to get task evidence. > - Small responses return inline JSON, so an agent must copy each document into a local helper. > - Copying changes bytes and uses model output for data transport. > - This pull request adds an explicit artifact export for non-sensitive GET reads. > - Helpers can download the exact response bytes and check their identity. ## Linked Issues or Issue Description **What existing behavior does this improve?** The native `call_api` response capture path. **Current behavior** Small JSON responses have no artifact reference. Local helpers need the model to copy the JSON data. **Proposed behavior** `saveResponse: true` saves a complete authorized GET response below or above the inline threshold. Zero-byte success returns an explicit empty receipt without storage. The receipt returns an artifact ID, byte size and SHA-256 for `download_artifact`. Other callers keep the current behavior. Related work: #17393 covers saved app-tool result size. #18105 covers mutation response paging. This change adds an explicit read export. ## What Changed - Add GET-only `saveResponse` validation and capture through the current artifact store. - Refuse sensitive operations, denied reads, partial responses and combined text paging. - Keep route authorization, actor visibility, storage budgets and download integrity checks. - Update the native tool contract, generated catalogs and API guide. - Add boundary and real-route byte comparison tests. - Preserve saved run state. This is an optional input field; old calls and receipts keep their current shape. No schema migration or identity reset is needed. The seeded catalog fixture is generated. ## Verification - Final published head `d47b52e1bf412418e24768f865cc139097a6d068`: [typecheck, build, all tests, Linux Rust and aggregate verify passed](https://github.com/paperclipai/paperclip/actions/runs/42918883477); [Windows Rust passed](https://github.com/paperclipai/paperclip/actions/runs/42918883348); [quality gate passed](https://github.com/paperclipai/paperclip/actions/runs/42918883016). Greptile completed successfully at this exact head, 5/5; all five review threads resolved. Automated Paperclip review remains queued due to a failed reviewer run; no final verdict is claimed for that service. - Current head: `d47b52e1bf412418e24768f865cc139097a6d068`. Merged current master to refresh GitHub's stale base snapshot; the PR now contains exactly nine intended artifact transport files, with no lockfile or UI changes. Fresh final-head CI/review is required. - Previous head `986ad87490fff93cfec313ba6fd56d3dc0bb665b` passed typecheck, build, Rust verification, artifact/native suites and completed test shards; remaining shards were still running when the base snapshot was refreshed. Greptile was 5/5, with one unrelated master UI finding resolved as outside the corrected PR diff. - Review fixes verified at `986ad87490fff93cfec313ba6fd56d3dc0bb665b`: 2,135 checks passed across the five focused suites; Runner TypeScript build and regenerated contract/catalog checks passed. This head replays the original implementation on current master and retains the original PR commit in its history. - Review red/green at replayed baseline `da3a1d4de9`: empty HTTP 200/204 tests failed with real storage's `File is empty`, then passed with zero new assets or quota reservations. Error-body tests failed without size/truncation fields, then passed. - Empty successes now return `empty: true`, `byteSize: 0`, `data: null` without storage. Nonempty JSON results such as `[]` export normally. Error receipts report byte size and truncation. Docs and tool guidance state the existing 10 MiB download limit and paged text alternative. - Previous-head CI typecheck/build/tests passed except the Rust process-lineage cleanup test (`unattributed_orphan`); final-head CI and fresh review remain required. - Red at `7368a0b15c`: the small-document export test failed with `Invalid call_api arguments` before production edits. The same check passed after the change. - Final post-rebase run at `6e7a35ed6a59bb74f4218312d9f10557099433c3`: all 2,124 checks passed across the API, capture, artifact boundary, real-route export and artifact download suites. - Real-route export test passed: HTTP bytes, stored bytes and downloaded bytes were equal. Foreign-company access and forged download metadata were rejected. - Existing API integration and artifact suites passed. Nearby API integration checks also passed (59 passed, 2 skipped). - Disposable native helper scan passed: 12 existing documents, 15 GET requests, zero mutations, 73,919 bytes, 17.985 seconds. All files passed size/hash checks. The bridge passed file references only. Production token savings were not measured. - Runner TypeScript build and generated contract checks passed. - `pnpm verify` returned exit 2 and required broader CI coverage. Local preflight stopped at missing Cargo after dependency and SDK setup. The local server typecheck stopped during the cycle layer without a diagnostic; the container recorded an OOM kill. This check has no pass verdict. Required CI must cover typecheck, build and test at the final head. ## Risks - Each explicit export uses the current capture budget and storage admission. Frequent small exports can reach that budget. - An artifact holds one HTTP response. Callers must still follow list pagination. - No database or saved-state migration is required. The option does not change default response behavior. ## Model Used OpenAI `gpt-6.1-sol`, through Codex in Paperclip Runner. Used reasoning, shell execution and native tools. The active agent configuration supplied this exact model ID. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |