mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-17 21:34:48 +01:00
90ae3c4337431a097bdb6327b211f3fbf77db59b
2300
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
90ae3c4337 |
fix(native): process a superseded export's late stop receipt and stop its sandbox when none arrives (#21143)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native run can be retired as superseded while its workspace export fails. Its files stay unsaved, and the run stays the writer of the task's workspace. > - #21136 records those files for the automatic workspace recovery. That record needs the sandbox's stop-and-retain receipt. > - The native run projection worker checked for that receipt for one hour after the run ended. After that, it dropped the run. Nothing woke it when the receipt arrived later. > - If the receipt came late or never, the run kept the task's workspace fenced. Every queued run on the task waited and then failed setup. Nobody owned the problem. > - This pull request makes the worker process a late receipt, stop and keep the sandbox itself when no receipt arrives, and never drop the run silently. > - The benefit is that the task's queue moves again by itself, and two writers never use one workspace. ## Linked Issues or Issue Description Refs #21136 (review thread on `native-superseded-workspace-export.ts:84`) **What happened?** `preserveSupersededWorkspaceExport` returned `recheckAt: null` when the sandbox stop receipt had not arrived within `finishedAt + 1h`. The projection worker then deleted its entry. A later stop receipt (`settleStopOnlyCleanup`) woke only the workspace recovery queue, not the projection queue. `preserveNativeFailedWorkspace` requires `hasConfirmedSandboxStopAndRetain`, so no hold was recorded. The run stayed the dirty writer, and queued runs kept failing setup. **Expected behavior** A late receipt is processed at once. When no receipt arrives, Paperclip stops and keeps the sandbox itself and waits for the provider's confirmation. The workspace stays held while the sandbox may still run. When the provider confirms that the sandbox is gone, the task continues from its last save. When the stop stays unconfirmed for hours, the task shows one plain notice, and the check continues. **Steps to reproduce** 1. Retire a native run as superseded after its workspace export fails (the #21136 shape). 2. Let the run's sandbox stop not settle (for example, the server that owned the release is replaced). 3. Wait more than one hour after the run ended. 4. See that the projection worker drops the run, and that a later stop receipt records nothing. The workspace stays held. ## What Changed - `sandbox-stop-and-retain.ts`: a confirmed stop-only receipt now also notifies `DELIVERY_QUEUES.nativeRunProjection` (scoped to the company and run). New `prepareUnconfirmedSourceSandboxStop` puts a stop-and-retain intent on the source lease from its current status. It skips a destroyed sandbox, a stop already in progress, and a cleanup in flight. - `native-superseded-workspace-export.ts`: - Within one hour of the run ending, the worker checks every minute (unchanged). - After that hour, it requests the stop-and-retain itself. The existing pending-cleanup sweep dispatches the stop and retries it until the provider confirms. The worker checks again every 15 minutes, and the receipt wakes it at once. The workspace stays held. - A sandbox that another lease holds is never stopped. - When the provider confirms that the sandbox was destroyed, the files cannot be saved. In one transaction (task, run, then workspace locks), the worker sets aside only that run's workspace epoch, wakes queued-run dispatch, and records `issue.previous_run_files_lost` on the task. - When the stop is still unconfirmed six hours after the run ended (or the sandbox is held by another lease), the task gets one `issue.previous_run_files_waiting` activity entry. The worker keeps checking. It never drops the run. - A stop-only dispatch now settles when the provider confirms that the exact sandbox is already gone. Before, the runtime accepted only a `stopped` reply and threw on `destroyed`, so a gone sandbox kept the stop pending forever. - `sandbox-stop-and-retain.ts`: new `acceptedStopOnlyReceipt` accepts `stopped`, or `destroyed` for a stop-and-retain intent (a native workspace export resume still needs `stopped`). `settleStopOnlyCleanup` stores a `destroyed` receipt as `released` / `success` / `workspace_source_unavailable`, removes the stop intent, never writes a retain receipt, and wakes the projection and workspace recovery queues. - `environment-runtime.ts`: the plugin `environmentStopLease` and built-in `stopLease` checks use `acceptedStopOnlyReceipt`. - `workspace-recovery.ts`: when its own stop settles as `destroyed`, the source is unavailable (the existing alternative-strategy path) instead of a binding failure. - Daytona plugin: stop of a sandbox that Daytona reports as not found returns `destroyed` (destroy already did this) instead of throwing. boxd already returns `destroyed`. Other sandbox plugins have no stop-only method, so nothing changes for them. - New shared `SANDBOX_ALLOCATION_HOLDER_STATUSES` (`active`, `retained`, `pending_cleanup`). The stop-only cleanup ownership check and the worker's "held by another lease" guard both use it, so a retained warm sandbox is never stopped and the two checks cannot drift. - `ui/src/lib/activity-format.ts`: plain labels for the two task activity entries. They do not ask a person to fix anything. - `doc/execution-semantics.md`: document the rule. - Not changed: the shared owner prover still refuses to treat a stopped instance on another host as proof that its child processes stopped. ## Verification - `server/src/services/superseded-workspace-export-recovery.test.ts` (4 new tests, red/green): - "checks again every minute while the run's own release may still stop its sandbox": the recheck branch. Nothing is recorded or stopped, and the workspace keeps its writer. - "stops and keeps the sandbox itself when no stop receipt arrived, and records the files once the late receipt lands": after the hour, the lease gets a stop-only intent. A second pass changes nothing. The late receipt wakes the projection queue. The next check records the hold, and the system recovery frees the workspace. Fails on master (`expected { …(19) } to match object { status: 'pending_cleanup' }`). - "names the unconfirmed stop on the task once, and keeps checking, when the provider never confirms it": the expiry branch. One notice, the check continues, no hold, and the workspace stays held. Fails on master. - "continues from the last save when the provider confirms the sandbox is gone": epoch 1 is set aside, the workspace is free, the task says so, and the result is not applied. Fails on master. - Review-round tests (all 5 fail on the first head `a208bfa58f`, pass now): - `environment-runtime.test.ts` "settles a plugin stop whose provider reports the sandbox gone (boxd, Daytona not found) through the cleanup sweep": provider dispatch through `releaseRunLeases` and `sweepPendingCleanupLeases`. The lease settles `released` with a `destroyed` receipt and no retain receipt. - `environment-runtime.test.ts` "settles a built-in stop whose provider reports the sandbox gone, without claiming retained files": the built-in `stopLease` path. No destroy call. - `superseded-workspace-export-recovery.test.ts` "continues from the last save when Paperclip's own stop finds the sandbox already gone": the worker's stop request gets a `destroyed` reply, the projection queue wakes, and the workspace is freed at epoch 2. - `superseded-workspace-export-recovery.test.ts` "never stops a sandbox a retained warm lease still holds, says so once and keeps checking": the source lease is unchanged, a recheck is returned, and `sandbox_in_use` is recorded. - Daytona `plugin.test.ts` "explicit stop confirms a sandbox Daytona no longer has as destroyed". - `environment-runtime.test.ts` (207) and Daytona `plugin.test.ts` (421) pass. - Related suites pass: `native-run-projection-work`, `retained-native-session-cleanup-work`, `workspace-recovery-stop-proof`, `native-failed-workspace-recovery`, `legacy-workspace-restore-recovery`, `environment-retained-orphan-sandbox`, `workspace-recovery-retry-copyback`, `workspace-recovery-fallback` (398 tests), and `ui/src/lib/activity-format.test.ts`. - `tsc --noEmit` for `server`. ## Risks - Low to medium risk. The new behavior acts only on an ended superseded run that failed in its workspace export, still holds a dirty workspace, and has no retained-file hold. - The worker now stops an ended run's sandbox one hour after the run ended if no stop receipt arrived. The stop uses the existing stop-only contract and keeps the sandbox. A sandbox held by another lease is never stopped. - A destroyed sandbox's unsaved files are not recoverable. The task continues from its last save and says so. - If a provider never confirms a stop, the workspace stays held (by design: no second writer). The task shows one notice, and Paperclip keeps retrying. - Shared cleanup code now treats a confirmed `destroyed` stop-only reply as settled. Every caller was checked: the cleanup sweep and run release count it as settled; the workspace recovery treats the source as unavailable; the fallback treats it as stopped (nothing can run there). No caller reads it as retained files, because no retain receipt is written. A native workspace export resume still requires `stopped`. - Each extra delivery notification per settled stop-only cleanup is cheap. The projection worker's scoped load finds nothing for runs that do not match. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (code search, shell, test execution) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
dfc510b887 |
fix(server): hand new run starts to the replacement server during a deploy overlap (#21127)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Paperclip Cloud replaces the server process on each release. The old server keeps serving until the new one passes health checks and takes the traffic. > - During that overlap the old server keeps starting new runs right up to SIGTERM, although the new server can already start them. > - SIGTERM then cuts off exactly those runs while they still prepare. In one audited swap the old server started 30 runs after the new server booted, and 23 of them ended `process_lost`. > - Admission must never pause, so the fix must hand over who starts new runs, not stop admission. > - This pull request makes the new server record when it starts runs. The old server then leaves new run starts to it, and starts them again if the new server stops or dies. > - The benefit is that a deploy no longer starts runs on a server that is about to stop, and the company always has one server that starts runs. ## Linked Issues or Issue Description Refs #20852 (preserves preparing and launching turns that a deploy interrupts; this PR reduces how many such turns the old server starts). Related: #21046, #21107 (shared controller-instance owner proof), #20986 (queued-run dispatch signal). **What existing behavior does this improve?** The start of new runs while two servers overlap during a Cloud deploy. **Subsystem affected** Server run admission (`server/src/services/heartbeat/queue.ts`), the controller instance table, and startup and shutdown in `server/src/index.ts`. **Current behavior** The old server starts queued runs until it receives SIGTERM. Runs that it starts in the last minutes of the overlap are interrupted while they prepare. Nothing tells the old server that its replacement already starts runs: `controller_instances.epoch` is written at process start, and that happens minutes before the new server can start runs. **Proposed behavior** The new server records `dispatching_at` and a short `dispatch_lease_expires_at` on its `controller_instances` row right after its startup queued-run pass. The old server reads this through an in-memory flag at run admission and leaves new run starts to the new server. Runs it already started continue and are handed over at SIGTERM as today. If the new server stops, drains or dies, the old server starts runs again within seconds. **Reason and benefit** Fewer runs are cut off by each swap. There is never a moment without a server that starts runs. **Breaking changes** None. The migration adds two nullable columns. Releases without this code ignore the columns, and their rows never carry the marker, so a mixed-version overlap behaves as today. ## What Changed - `controller_instances`: new nullable columns `dispatching_at` and `dispatch_lease_expires_at` (migration `0471_controller_instances_dispatch_marker`, two `ADD COLUMN` statements, no lock beyond the catalog update). - New `server/src/services/run-start-handover.ts`. It is one worker on the shared delivery coordinator (`DELIVERY_QUEUES.runStartHandover`). Its renewal and recheck are coordinator deadlines, and a drain change or expiry wakes it. - Replacement side: `begin()` records the intent and writes the marker (it creates the boot's row if the row is missing). If the server drains at that moment, the worker writes the marker when the drain ends. The worker then publishes a `run-start-handover` hint and renews the 20 s lease every 5 s. It renews only while an older boot that is not stopped or fenced may still serve, and for at most 30 min. All marker writes run in one serialized chain. `release()` at shutdown lands after any renewal already in flight. A clear that fails is retried, and shutdown waits for the release. While the server cannot dispatch (drain, quiesce, admission closed), the worker clears the lease. - Old-server side: `isYielding()` is an in-memory read. The hint refreshes it, and so does one read that a dispatch pass waits for (at most 1 s) when the last read is more than 2 s old. A read that gets no answer within 1 s never keeps the server yielding. The server admits, and the yield also ends when the replacement's last seen lease runs out on the old server's own clock. Reads are a single flight on the dedicated connection. While it yields, the server rechecks every 3 s, calls `onResume` when the marker is gone, and sends a coalesced queued-run-dispatch hint for skipped starts. - It yields only to a newer epoch with a live marker. It never yields to an older boot, to itself, or to a row without the marker, and it does not yield without a row of its own. - **Database invariant:** every statement of the hand-over runs in a small transaction on the dedicated one-connection controller-lease pool (`controllerLeaseDb`) with a 1 s statement timeout. That covers reads, marker writes, the `run-start-handover` notification (commit-bound, in the marker write's own transaction) and the skipped-start hint. None of it touches the shared pool or registers tracked database work. The pool is shared only with legacy controller-lease renewals, and never with the liveness pool that native lease renewals use. A hand-over statement waits at most one legacy batch (10 s), inside the 15 s between a marker renewal and the marker's expiry. It delays a legacy renewal (60 s lease, renewed every 10 s) by at most 1 s. I did not piggyback on the controller-instance lease renewal, because that renewal runs only while the server holds work and only every 30 s. - **Import audit of `run-start-handover.ts`:** - `drizzle-orm` `sql`: builds statements, no I/O. - `controllerLeaseDb`: the dedicated connection, used by every statement here. - `notifyDatabaseWorkOnCommit` (new in `work-signals.ts`): runs `pg_notify` only on the transaction it is given and registers no intent, so the coordinator's `reconcileDatabaseWork` has nothing of ours to check. - `logger`: no I/O to the database. - `controller-process-identity`: in-memory boot id and host hash. - `delivery-work-coordinator`: type only. The worker is registered for lifecycle and deadlines; its `run` awaits only the serialized marker chain, and reads are not awaited. - `DELIVERY_QUEUES`: constants. - `task-admission` drain subscriptions: in-memory events. - `heartbeat.ts`: run admission is also closed while the server yields. A `runStartHandover` service option (default: the process's registered hand-over) lets one process host two assembled services in tests. `queue.ts`: `startNextQueuedRunForAgent` and `resumeQueuedRunsAfterDrain` first call `refreshRunAdmission()`. That call returns `null` (nothing to await) unless the hand-over read is stale. - `index.ts`: creates the hand-over only when the instance lease is on, the deployment notification channel exists, and the scheduler is enabled. It calls `begin()` after the RunDispatch startup pass and wakes that queue. It releases the hand-over when run admission closes at shutdown, and stops it before the lease stops. - `deployment-notifications.ts`: keeps the LISTEN connection open while a hand-over involves this server, so skipped starts and lease changes reach the other server at once. - `doc/execution-semantics.md`: new section "Run starts during a deploy overlap". It covers the marker, who yields, failure handling, what counts as starting a new run, and what a post-deploy audit should show. **Marker contract for other components (for example the deployer):** a replacement starts runs when its `controller_instances` row has an `epoch` greater than the old boot's, and `dispatching_at is not null and dispatch_lease_expires_at > now() and stopped_at is null and fenced_at is null`. `dispatching_at` stays set after the lease lapses, as the time the boot began starting runs. ## Verification - `cd server && npx vitest run src/__tests__/heartbeat-run-start-handover.test.ts`: assembled-app test with two `heartbeatService`s, each with its own pool, listener, coordinator and boot. A wake that the old server accepts stays queued there. The replacement's real queued-run dispatch starts it, and claims are observed per pool. After a replacement release, the old server starts the run itself. Red: with the yield removed from `isRunAdmissionClosed`, the old server claims the run. - `cd server && npx vitest run src/__tests__/run-start-handover.test.ts`: 19 tests pass against embedded PostgreSQL. Each test uses two servers, each with its own pool, boot id, LISTEN connection and delivery coordinator. The tests cover: - the old server is ready and the new server starts; the new server sets the marker; the old server yields; the queued run is claimed only by the new server, and a skipped start wakes the new server's dispatch - the old server learns from its own dispatch path without a notification: a read older than 2 s is refreshed before the start - the new server dies after it sets the marker: the old server resumes and runs its queued-run pass - the new server shuts down: the old server resumes at once - the new server drains: it releases the marker and takes it back afterwards - a replacement that starts up during a drain takes over when the drain ends - a shutdown release during an in-flight renewal stays released - a clear that fails at shutdown (statement timeout behind a row lock) is retried by a later release - an older server that outlives the lease TTL keeps yielding while the replacement renews - a yield that the old server cannot confirm within 1 s (its only connection is busy) does not stop starts (red/green) - a yield ends when the replacement's lease runs out, even while every read waits for a connection (red/green) - renewals blocked past the TTL hand starts back once, then the replacement takes them again (bounded flapping) - a clear that fails when the replacement starts draining is retried well inside the lease (red/green) - with both servers' shared pools blocked past the TTL, the marker is written, renewed and released, and the old server yields and resumes, all before the block ends (red when hand-over statements use the shared pool) - an older-release row without the marker changes nothing - a server never yields to an older boot or to itself, and does not yield without a row of its own - a lone server writes once, never renews, and admits from memory - without a hand-over (self-hosted) there is nothing to await - Red: on `origin/master` the test file cannot run, because the module and the columns do not exist. The old server has no signal and keeps starting the handed-over run. - `cd server && npx vitest run src/__tests__/deployment-notifications.test.ts src/services/heartbeat/dispatch-work.test.ts src/__tests__/delivery-work-coordinator.test.ts src/__tests__/controller-instance-lease.test.ts src/shutdown.test.ts`: all pass. - `cd packages/db && pnpm check:migrations`: safety and compatibility checks pass. `cd server && npx tsc --noEmit -p .`: clean. - `pnpm verify` asks for broad verification because `doc/execution-semantics.md` is a global CI input. Required CI supplies that coverage. ## Risks - Run starts move from the old server to the new server during the overlap. If the new server cannot dispatch, the old server resumes on the lease clear (about 3 s) or the lease lapse (up to about 23 s after a crash). If a hand-over read fails, the server keeps admitting. - If the old server was not listening for notifications, it can start runs for up to about 2 s after the marker (the age of its last read). On a dispatching server, the pre-dispatch read adds one small bounded query at most every 2 s. - An older boot that has been idle for more than 30 min counts as gone. The new server then does not renew, and the old server admits again as today. - The migration adds nullable columns only. The deployer can apply it while the previous release serves. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits) and extended thinking. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
2174f2dfa5 |
perf(server): stop re-reading and re-decrypting unchanged checkpoint headers on every upgrade readiness probe (#21130)
Cloud repeats the upgrade readiness probe. Every probe loaded, decrypted and parsed the checkpoint header of each active native run and of each retained task session's last run, even when nothing had changed. The statement that reads a run now also reads its header's row revision (xmin, ctid, updated_at, stored size) and returns the material only for a revision the process has not inspected. A bounded memo in the checkpoint owner keeps the four identifiers the answer compares. It reuses an answer only for the same binding, row revision and secrets key, loads the key again for each answer it serves, never remembers a header that failed to decrypt, and forgets a header the latest probe did not inspect. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
f2e449f8f6 |
fix(retry): continue an accepted Retry past chat input held because it predates the stop (#21129)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A person's Retry on a connected task (for example a GitHub pull request review) must start work. #21080 made a Retry whose chat replay is refused after the click continue from the recovered saved work > - After #21080 was deployed, accepted Retries on a busy GitHub review agent still did not start > - Those tasks had a saved chat request that arrived while the failed run was still running. Admission holds such a request as `message_predates_stop`, so it can never start alone > - #21080 refused to continue while any saved chat input existed, so this held input blocked the continuation. For a second group, the latest run was an admission that the recovery hold cancelled before dispatch, and the recovered saved work belongs to the earlier failed run > - This pull request lets the continuation adopt input that was held because it predates the stop, and lets the recovered run that held a cancelled admission count as the saved work > - The benefit is that an accepted Retry starts work on these tasks too ## Linked Issues or Issue Description Follow-up to #21080. No public issue exists. This is the bug description, in the fields of `.github/ISSUE_TEMPLATE/bug_report.yml`. **What happened?** After #21080 was deployed, `POST /api/issues/:id/retry` returned 202 for blocked tasks of a GitHub review agent (`paperclip_runner`), but no run started. Each Retry receipt changed to `failed` about 5 seconds later (on the 14:48 Retries, 5 seconds for one task and about 75 seconds for two others), with no run and no `issue.retry_continued_from_saved_work` activity. Every affected task had a saved chat request (a newer GitHub review request) with an `issue.execution_waiting` record of reason `message_predates_stop`. The request arrived while the failed run was running, so admission holds it permanently. The interruption projection kept pointing at that held request. In a second group of tasks, the latest run was cancelled before dispatch with `execution_reconciliation_required`, and the recovered saved work belonged to the earlier failed run that held it. Root cause: in `continueRefusedTaskChatRetry` (`server/src/services/task-retry.ts`), any saved chat input from `selectTaskRetryInput` stopped the continuation, so the receipt settled `failed`. The held input can never start by itself, so nothing ran. The recovered-work check also looked only at the latest run, so it missed the recovered run that held a cancelled admission. **Expected behavior** An accepted Retry starts work. Saved input that admission holds because it predates the stop joins the continuation, and the task continues from the recovered saved work. **Steps to reproduce** 1. Bind a task to a GitHub chat conversation. Let a native run fail with `native_continuation_requires_reconciliation`, and recover its saved work. 2. Save a chat request while that run is still running, so admission holds it with `message_predates_stop`. 3. Click Retry while the chat identity lock is busy, so the source check is deferred and then refused. 4. The route returns 202. The Retry receipt changes to `failed`, and no run starts. **Paperclip version or commit** |
||
|
|
df89950eb6 |
fix(native): save or set aside a superseded run's unsaved workspace so its task queue moves (#21136)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native run saves its accepted result. Then a finalizer exports the run's workspace files and applies the result to the task. > - When a newer run already decided the task, the finalizer retires the older result as superseded. Since #21106, this also ends the run. > - If that retirement interrupts a failed workspace export, the run's files stay unsaved. The run also stays the writer of the task's workspace. Nothing records those files for recovery, so nothing ever releases the workspace. > - Every run queued on the task then waits behind a save that never settles. After one hour, each queued run fails setup instead. > - This pull request records those files for the existing automatic workspace recovery. Recovery saves them from the retained sandbox, or, when every bounded attempt fails, continues the task from its last save with the files attached. > - The benefit is that the task's queue moves again by itself, and the run's unsaved work is saved or kept findable. The superseded result is never applied. ## Linked Issues or Issue Description Refs #21106 **What happened?** On a hosted staging server, a native run accepted its result at 11:31:08Z. Its workspace export failed once with `remote_workspace_archive_fs_eacces`. The finalizer then retired it as `native_finalization_superseded`, because another run had already decided the task. #21106 ended that run at 14:59:17Z (`failed`). The task still could not move. The next queued run was cancelled with `workspace_busy` ("ended, but its workspace save has not settled"). Its retry re-armed every minute. Four more runs stayed queued. The task showed no recovery action. **Expected behavior** The ended run's unsaved files are saved from its retained sandbox, and the workspace is released. If they cannot be saved within the bounded retries, the task continues from its last committed save, and the files are attached to the task. The queued runs proceed in both cases. The superseded result is never applied. **Steps to reproduce** 1. Accept a native result for a run that writes a remote workspace. 2. Let the workspace export fail once, while a newer run has already committed the task's decision. 3. See that the finalizer is `terminal_failure` / `native_finalization_superseded` and the run is `failed`. 4. See that the workspace still names the run as its dirty writer, and that no retained-file recovery action exists. 5. Queue another run on the task. It defers with `workspace_busy`, and after one hour it fails setup with "The workspace still has an unfinished writer or save operation." **Root cause** `recordRetryableFailure` (`native-run-finalizer.ts`) retires a superseded finalizer without preserving the export. `preserveNativeFailedWorkspace` records a retained-file hold for an accepted result only when the failure code is `native_workspace_sync_out_retry_exhausted`. A `native_finalization_superseded` coordinator never qualifies. `readRemoteWorkspaceWriterHold` keeps an ended dirty writer as the holder until its save settles. Without a hold, no worker ever settles it. **Paperclip version or commit** Staging Core with #21106 (`81d2e4d8d9`). Master `ef2b53b675` has the same behavior. ## What Changed - `native-superseded-workspace-export.ts` (new): identifies a superseded finalizer that failed in the workspace export. It selects ended runs that still hold their workspace with unsaved files and no retained-file hold. It records the hold through `preserveNativeFailedWorkspace`. It checks again only while a just-ended run's sandbox stop receipt is still arriving (at most one hour). - `native-failed-workspace-recovery.ts`: `preserveNativeFailedWorkspace` treats a superseded workspace export like an exhausted copy-back, with the same proofs (run ended, no live controller, accepted result, confirmed sandbox stop-and-retain, source still bound to the workspace). It marks the hold `supersededWorkspaceExport`. - `native-run-projection-work.ts`: the native run projection worker also selects these runs. It loads them at startup and wakes when a finalizer changes. Rows written by older releases are therefore settled after deploy without a person. - `workspace-recovery-fallback-discovery.ts` and `workspace-recovery-fallback.ts`: a marked hold whose bounded re-export is spent (at most 5 retries in 30 minutes) is eligible for the existing continue-from-last-save fallback. That fallback sets aside only this run's workspace epoch and keeps its location on the workspace record. It attaches the readable files to the task unapplied. The task message says Paperclip "could not save" the previous run's unsaved files. - `preserved-result-finalization.ts`: a superseded result is never resumed for application, even after its files are saved. - `doc/execution-semantics.md`: document the rule. - No schema change. Nothing changes on the path that starts a run or a turn. ## Verification - `server/src/services/superseded-workspace-export-recovery.test.ts` (new). The first two tests fail on master and pass on this branch: - "records the files for automatic recovery, saves them from the retained sandbox and frees the workspace". It uses the staging row shape. The projection worker records the hold, and a second pass changes nothing. The workspace waits for that save. The system recovery saves `saved.txt` into a committed snapshot. The workspace becomes free. The finalizer, the result and the task's decision do not change, and the result is not resumed. On master the worker selects nothing (`expected 0 to be 1`). - "continues from the last save with the files attached when every re-export fails". After an exhausted re-export budget, the hold is resolved `restored`. The files are attached (`quarantine.state = captured`). The workspace moves to epoch 2 with no writer, and the abandoned epoch keeps its lease location. The message has no request for a person to fix anything. - "leaves a superseded result alone when it never failed in the workspace export". - Related suites pass: `native-run-projection-work`, `native-finalization-recovery`, `workspace-recovery-fallback`, `native-failed-workspace-recovery`, `workspace-recovery-retry-copyback`. - `tsc --noEmit` for `server`. ## Risks - Low to medium risk. The change acts only on runs whose finalizer is already terminal and superseded, that failed in the workspace export, and that still own a dirty workspace. All existing preservation proofs still apply. - The saved files of a superseded run become the workspace's next saved version. The run's result (the task decision) is still never applied. - When the export can never succeed, the task continues from its last save. Files that cannot be read are not attached. The workspace record keeps the set-aside epoch's location. Saving the readable part of a partly unreadable workspace is not in scope. - Runs that already failed setup before this deploys are not restarted by this change. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (code search, shell, test execution, read-only API inspection) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
782d6bead4 |
Configure managed plugin Runner lifecycle (#21065)
## Thinking Path > - Paperclip runs agents in managed environments. > - An agent can use an explicit environment or the instance default. > - A lifecycle plugin must receive the driver that provisions the agent. > - A managed plugin environment also needs to select the Runner process lifecycle. > - This change gives lifecycle selection and provisioning the same environment lookup. ## Linked Issues or Issue Description **What existing behavior does this improve?** Managed plugin environment provisioning and lifecycle configuration. **Current behavior** Lifecycle calls lack the selected plugin driver. Managed plugin configuration cannot select warm Runner execution. **Proposed behavior** Resolve the environment once per lifecycle operation. Pass the driver key for the plugin that serves it. Parse and persist the managed Runner lifecycle mode. The portable ownership and workspace placement changes are in [#21066](https://github.com/paperclipai/paperclip/pull/21066) and [#20743](https://github.com/paperclipai/paperclip/pull/20743). ## What Changed - Share agent environment resolution between required plugin selection and the lifecycle RPC. - Pass `environmentDriverKey` to the plugin that serves the selected environment. - Allow managed plugin environments to set `warm` or `per_turn` and persist the selected mode. - Advertise `pluginEnvironmentRunnerLifecycleMode` in the release capability manifest so senders can check support before including the field. ## Verification - Built shared types, the plugin SDK, and Runner. Server typecheck and SDK source-pin check passed for the initial layer. - The capability contract test failed before the manifest fix and passed after it. Independent verification passed all 160 tests across managed configuration, plugin environment persistence, lifecycle routing, and environment reconciliation. - Routing coverage includes the instance default, an explicit second driver from the same plugin, and an explicit sandbox. - [Hosted CI](https://github.com/paperclipai/paperclip/actions/runs/43193311862) passed on `f07b192c21`. The first attempt timed out in an unchanged workspace-recovery fixture and then collided with a background recovery action. The exact test and all 44 tests in its suite passed locally; the isolated shard retry passed without code changes. ## Risks - Existing managed configuration can omit the lifecycle mode. Senders must check the target release for `pluginEnvironmentRunnerLifecycleMode` before sending the new field. Publishing this signal does not update external deployment tools. ## Model Used - OpenAI Codex GPT-6 with GPT-6.1 Sol agents for implementation and independent review. ## Checklist - [x] Thinking path and model are included. - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] The issue is described using the enhancement template. - [x] Local checks passed and new behavior has tests. - [x] All Paperclip CI gates are green. - [x] Greptile is 5/5 with no open findings. --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
1af987690d |
perf(storage): hash and encrypt workspace snapshots off the server thread (#21138)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - One server process serves every request and agent turn on one thread. The same process saves each remote workspace as one encrypted archive in object storage, and restores it before a turn > - `server/src/services/remote-workspace-snapshot-storage.ts` hashed, encrypted and decrypted every byte of those archives on that thread. `server/src/storage/s3-multipart.ts` computed the Content-MD5 of each uploaded part on that thread too > - Hashes and ciphers are synchronous in Node. A CPU profile of a busy hosted server showed this work on the main thread, which was at 93% of one core > - Each megabyte of a snapshot delays every request and turn that waits behind it > - This pull request moves those passes to a small, bounded set of worker threads. The checks, their order, their failures and the stored format do not change > - The benefit is that a 1 GiB save uses about 0.7 s of the server's thread instead of about 2.8 s, and an unrelated request during the save is answered in 1.3 ms (p99) instead of 16.7 ms ## Linked Issues or Issue Description No public issue exists. I searched open and closed pull requests for `workspace snapshot hash encrypt`, `event loop` and `archive io`. Related merged work: #19900 gave these stream passes 1 MiB buffers, #20554 removed a read-back from migration, #20722 took runner artifact hashes off the event loop, #20709 records long event-loop blocks. No open pull request covers this change. **What happened?** A 30 second CPU profile of the main thread of a hosted server (commit |
||
|
|
5b698b68cf |
perf(run-events): append a native event in one transaction of two statements (#21131)
One appended native run event cost eight statements on the database connection (BEGIN, SAVEPOINT, run lock, replay probe, sequence update, insert, count/max read, COMMIT). The query builder built five of them again for every event, and each event created two transaction objects. A native source event now takes the run lock and then runs one statement that finds a replayed event or reserves the sequence and inserts, and reads the retained count and maximum for the acknowledgement. The lock stays a statement of its own so the append reads a snapshot taken after the lock was granted. The control-plane port passes its controller check to the append instead of opening an outer transaction, so there is no SAVEPOINT. The hot statements are fixed SQL text. Events that wake a queue still register the wake before the write and only when the event is new. Order, sequence numbers and durability do not change: every event still commits before its append returns. Local benchmark (server/scripts/benchmark-run-event-append.ts), one native event: 8.3 -> 4.6 statements, 589 -> 222 KB allocated, 1.09 -> 0.53 ms CPU, 28 -> 11.5 garbage collections per 1,000 events. One ordinary event: 105 -> 27 KB allocated. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
5c953c1c1b |
fix(ui): keep chat pins first and allow drag reordering (#20830)
Keep pinned chat agents first in saved order. Add pointer and keyboard reordering with shared navigation pin controls. Preserve focus and hidden pin positions. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
|
ef2b53b675 |
fix(environments): keep resuming an idle sandbox whose resume timed out (#21113)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A task that runs on a reusable cloud sandbox resumes the same sandbox on the next wake > - When the resume call times out, the run failed after 90 seconds with `reusable_sandbox_resume_failed (cause=transport_timeout)` > - The provider keeps starting the sandbox after the host call times out, so the sandbox is usually ready a little later > - The run gave up after one call, and the task then waited behind a Retry button > - This pull request keeps the same run retrying an idle sandbox whose resume only timed out, up to the provider's start deadline > - The benefit is that recovery works automatically and a sandbox with unsaved work is never destroyed ## Linked Issues or Issue Description Refs #20626, #20828, #21020 **What happened?** Three Codex runs on Daytona failed before provider dispatch with: `Reusable sandbox lease "<id>" could not be resumed (cause=transport_timeout); the lease was preserved and no replacement was created.` Each run had `executionRecovery: { kind: "bootstrap", providerWorkStarted: false }`. Each run took 100 to 122 seconds: one resume call used the whole 90-second budget, so no in-run retry was possible. The preserved leases were idle. Their last runs had ended about two hours before, with a confirmed stop-and-retain. The workspaces were not safe to replace, so the run failed strictly. The automatic bootstrap retry was then refused ("The automatic retry could not revalidate the original user continuation."), and all three tasks ended `blocked` behind a Retry. **Expected behavior** A resume that only timed out, before any provider work, on a sandbox that no live run holds, continues in the same run. The run keeps the same sandbox and does not create a replacement or destroy the sandbox. **Steps to reproduce** 1. Let a task's reusable Daytona sandbox stop and stay idle for some hours (it auto-archives after 60 minutes). 2. Retry the task. The workspace has unsaved work, so replacement is not safe. 3. The `environmentResumeLease` call takes longer than 90 seconds. The host call times out (`RPC call "environmentResumeLease" timed out after 90000ms`), the 90-second budget is spent, and the run fails with `reusable_sandbox_resume_failed`. **Paperclip version or commit** Staging at `849f109f` (current `master` has the same code path). **Deployment mode** Hosted staging (cloud), Daytona sandbox provider plugin, reusable leases on. **Agent adapter(s) involved** `codex_local` (Codex) on a Daytona sandbox environment. The change is adapter-independent. **Root cause** `server/src/services/environment-runtime.ts`, reusable resume loop: the first `environmentResumeLease` call got the whole 90-second budget as its RPC timeout. When that call timed out, the loop saw no time left and threw. The 90-second budget assumes that the bounded bootstrap retry resumes the same sandbox later. That retry is not always admitted. A timeout is different from a refusal: the provider is still starting the sandbox. ## What Changed - `reusable-sandbox-resume-replacement.ts`: add `reusableSandboxLeaseIsIdle` (lease is `released` or `retained` and no other live run holds it). `reusableSandboxReplacementIsSafe` now uses it, with the same rules as before. Add `REUSABLE_SANDBOX_IDLE_RESUME_BUDGET_MS` (330 seconds, the Daytona start deadline plus the call margin). - `environment-runtime.ts`: when the 90-second budget ends on a `transport_timeout` and the preserved lease is idle, extend the deadline one time, up to the provider start deadline (at most 330 seconds from the first attempt). The run keeps retrying the same sandbox with backoff. Each call still has at most 90 seconds. Refusals, a live owner, and other failures keep the current behavior. The run never replaces or destroys the sandbox because of this extension. - `doc/execution-semantics.md`: document the idle-timeout extension. ## Verification - Red before the fix, with the new tests in `server/src/__tests__/environment-runtime-reusable-resume-replacement.test.ts` (staging shape: released lease, finished owner run, workspace with unsaved work, 300-second provider deadline, host RPC timeout): 2 tests failed with the staging error `could not be resumed (cause=transport_timeout); the lease was preserved and no replacement was created.` - Green after the fix: 16/16 pass. The timed-out idle sandbox resumes in the same run with no create or destroy call. If every call times out, the run stops at or before 330 seconds, makes more than one call, and keeps the lease `released`. A sandbox that a live run still holds gets one call and fails at 90 seconds, as before. - `pnpm exec vitest run src/__tests__/environment-runtime.test.ts src/services/environment-runtime-reusable-lease-probe.test.ts src/services/remote-workspace-transport.test.ts`: 378 pass. - `tsc -p server/tsconfig.json --noEmit` passes. - `pnpm verify` exits 2 because a doc is a global CI input. Required CI supplies the broad coverage. - Not run: a live Daytona resume on staging. No later run attempted the three preserved leases, so it is inferred, not observed, that the sandboxes were ready after the host call timed out. ## Risks - A run can now wait up to 330 seconds in preparation before its first turn, but only when the resume timed out on an idle sandbox. Before this change, that run failed and the task waited for a person. - A provider that is down (not slow) now takes up to 330 seconds to fail instead of 90 seconds. The failure is the same `reusable_sandbox_resume_failed`, and the lease is still preserved. - No persisted fields change. Leases saved by older releases follow the same rule. - The refused bootstrap retry admission for task Retry continuations is a separate gap. This pull request does not change it. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
81d2e4d8d9 |
fix(native): end a run whose finalizer a newer run superseded, so it stops holding its task (#21106)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native run saves its accepted result, then a finalizer exports the workspace and applies the result to the task. > - When a newer run already committed the task's decision, the finalizer retires the older result as `native_finalization_superseded`. > - That retirement set the finalizer terminal, but it did not end the run. The run stayed `running` and held the task and every run queued behind it, with no way out except a person. > - A stuck run must never hold a task and its queue forever. > - This pull request ends the run when its finalizer is retired as superseded, and it ends runs that an older release left in that state. > - The benefit is that the task continues with its queued runs by itself. ## Linked Issues or Issue Description Refs #21046 **What happened?** On a hosted staging server, a native run accepted its result at 11:31:08Z. The workspace export then failed (`remote_workspace_archive_fs_eacces`). The task's last decision came from another run that was created 42 seconds later but ran earlier. The finalizer judged this run superseded: it set the finalizer to `terminal_failure` with `native_finalization_superseded`. The run stayed `running` with "Recovery needed". Over one hour later it still held the task, and five queued runs waited behind it. A server upgrade at 12:02Z did not change it. **Expected behavior** A run whose finalizer is retired ends. The task keeps the newer run's decision, the retired result is never applied, and the queued runs proceed. **Steps to reproduce** 1. Accept a native result for a run. 2. Commit a status decision on the same task from a run that was created later. 3. Let the older run's finalization fail once, so the finalizer records the failure. 4. See that the finalizer is `terminal_failure` / `native_finalization_superseded` and the run is still `running`. **Root cause** `recordRetryableFailure` (`server/src/services/native-runtime/native-run-finalizer.ts`) projects a terminal run status only when `exhausted && !supersededByNewerRun`. A superseded retirement therefore never ended the run. No other path ends a run whose finalizer is already `terminal_failure`: the reconciler skips `terminal_failure` rows. **Paperclip version or commit** Staging Core `3eca7a6112`. The behavior is the same on master `482063aeca`. ## What Changed - `native-run-finalizer.ts`: a superseded retirement now ends the run with the same status mapping as an exhausted finalization (workspace failure: `failed`; otherwise the provider's terminal state). It does not change the task, does not apply the result, and does not keep the workspace export lease. The run's error says in plain words that a newer run already decided the task. - `native-run-finalizer.ts`: add `endSupersededNativeFinalizationRuns`. It ends a native run that is still `running` while its finalizer is `terminal_failure` / `native_finalization_superseded`. This covers rows that an older release wrote. It locks in the finalizer's order (coordinator, task, run), and re-checks the row under the lock. - `native-run-projection-work.ts`: the registered native run projection worker (it loads at startup, pages through all candidates, and wakes on finalization changes) also selects such runs and ends them with `endSupersededNativeFinalizationRuns`. - `doc/execution-semantics.md`: document the rule. - No schema change. Nothing changes on the path that starts a run or a turn. ## Verification - `server/src/__tests__/native-finalization-recovery.test.ts`, red on master and green on this branch: - "retires an older failed finalizer when a newer run already committed the issue": the retired run now ends (`succeeded`, `finishedAt` set). Master: the run stays `running`. - "ends a run an earlier release left running after retiring its finalizer as superseded": the shape of the staging row (workspace export failed, superseded finalizer, run `running`). The repair ends it as `failed` with the plain message. The task, its decision and the finalizer stay unchanged, and no status decision is written for the run. A second pass changes nothing. - `server/src/__tests__/native-run-projection-work.test.ts`, "ends, at startup, a run an earlier release left running after retiring its finalizer as superseded": the registered worker ends the staging-shaped row at startup and settles its claimed wake. Red without the worker change. - Related suites pass: `native-finalization-recovery`, `native-run-projection-work`, `native-finalization-reconciler-timing`, `native-reconciliation-timing`, `native-workspace-finalizer`, `native-workspace-finalizer-failure`, `native-run-finalizer-telemetry`. - `tsc --noEmit` for `server`. ## Risks - Low risk. The change only ends runs whose finalizer is already terminal and superseded. It never applies a result and never changes the task. - A superseded run whose workspace export failed now releases its sandbox lease through the normal settlement instead of waiting forever. Its result was not applied, because a newer run decided the task. - Not changed here: the superseded check compares run creation times. A run created earlier but executed later is still judged superseded. This PR makes that verdict end the run; it does not change the verdict. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (code search, shell, test execution, read-only API inspection) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
0a2f618cae |
fix(native): leave a run another live server is preparing alone, and re-adopt a run a shutdown handed off (#21107)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - An upgrade replaces the server. The old and the new server serve together for a few minutes, then the old server gets SIGTERM and hands its native runs to the next controller. > - #21046 taught the legacy reaper to leave a run with its owner while the owner's server process is proven alive. Native runs have a similar gap. > - Runtime selection writes a native run's coordinator with no controller and no lease, and stops the run lease. The executor claims the coordinator about 20 seconds later. In that window the other server's sweep sees no owner and marks the run "ownership could not be verified". > - When the old server then hands the run off before any Runner checkpoint exists, no sweep looks at it again. Only the next server boot does, 15 to 40 minutes later. The task and its queued runs wait. > - This pull request leaves such a run with its live preparing server, and lets the serving server re-adopt a handed-off run through the ordinary restart claim. > - The benefit is that a run started during an upgrade continues within one sweep, with no false "blocked" mark and no wait for another upgrade. ## Linked Issues or Issue Description Refs #21046 **What happened?** On a hosted staging server, an upgrade started a new server at 12:19:57Z. The old server started a native run and selected its runtime at 12:20:32Z. At 12:20:35Z the run logged "Native execution ownership could not be verified; automatic recovery is blocked" (`observed_owner_unverified`). The old server's executor claimed the run at 12:20:51Z. The old server got SIGTERM and suspended the run at 12:20:54Z (`awaiting_runner_reattach`). After that the run showed "Confirming execution" for 29 minutes, and two queued runs waited behind it. At 12:49:45Z the next upgrade's startup recovery restarted its bootstrap, and the run succeeded at 12:58Z. **Expected behavior** - A server does not mark a run unverified, or recover it, while another live server is preparing it. - A run that a graceful shutdown hands to the next controller is re-adopted by the serving server at once, not at the next boot. **Steps to reproduce** 1. Seed a native run whose coordinator is `observed` with no controller and no lease, and whose `controllerProcess` names another server with a live controller instance lease. 2. Run the recovery sweep on this server. See that the run gets `native_execution_ownership_unverified`. 3. Claim the coordinator for the other server, then write what its graceful shutdown writes (`restart_suspended`, `awaiting_runner_reattach`, lapsed lease) and stop its instance lease. Give the run a predecessor's Runner PID. 4. Run the sweep again. See that nothing claims the run. **Root cause** - `recoverActiveExecutions` (`server/src/services/heartbeat/recovery.ts`) treats an `observed` coordinator without a live controller lease as an owner it cannot verify. Between runtime selection and the claim, the only owner on record is the run's `controllerProcess`, and the sweep did not check it. - `listExpiredNativeRunnerAuthorities` (`native-runner-checkpoint.ts`) selects runs for the restart claim only when they have a controller checkpoint, or no process identity at all. A run handed off just after its claim has no checkpoint, and a follow-up carries its predecessor's Runner PID. The sweep then skips `awaiting_runner_reattach` runs, so only a server boot looked at them again. **Paperclip version or commit** Staging Core `814fe5d20f` (old server) and `3eca7a6112` (new server). The behavior is the same on master `849f109f62`. ## What Changed - `recovery.ts`: before it lists expired native runs, the sweep checks runs in the runtime-selection window (coordinator `observed`, no controller, no lease, not tracked here). When the run's `controllerProcess` names another server process that the shared owner prover proves alive, the sweep leaves the run with that server: it does not recover it and does not mark it unverified. It re-checks within 30 seconds. After 5 minutes without a claim (`LEGACY_LIVE_OWNER_SETTLE_GRACE_MS`), or once that server is gone, the existing rules apply. - `native-owner-stamps.ts`: add `preparingControllerOwnerStamp` for that proof. - `native-runner-checkpoint.ts`: `listExpiredNativeRunnerAuthorities` also selects a run that a graceful shutdown handed off and whose lease lapsed: recovery state `awaiting_runner_reattach` or `awaiting_evidence`, with a `restart_suspended` history entry that no restart claim followed. It stays selected through ownership waits (the departing server can still be alive at the first look). `recovery.ts` schedules a 30-second re-check for an ownership wait that has no retry of its own. The ordinary restart claim then proves the takeover and picks the disposition, exactly as the startup pass of a new server does. A row that an older release already marked unverified is re-adopted too: the claim clears the mark. - `doc/execution-semantics.md`: document both rules. - No schema change. Nothing changes on the path that starts a run or a turn: the new proof runs only in the recovery sweep, and only for runs in the window. ## Verification - `server/src/__tests__/heartbeat-native-restart-suspension.test.ts`, block "a run the old server started during an upgrade overlap". Red on master, green on this branch: - "leaves the run with its live preparing server, then re-adopts it after that server hands it off": two concurrent sweeps leave the run `running` with no error and do not touch the coordinator. After the old server claims, hands the run off and stops, one sweep of the new server re-adopts it and reaches provider dispatch. Master: the run is marked `native_execution_ownership_unverified`, and after the handoff nothing claims it. - "keeps a handed-off run eligible while the departing server is still alive, and re-adopts it once it exits": the first sweep waits (`awaiting_evidence`), the sweep after the departing process exits re-adopts the run. Red with a filter that matches only the last history entry. - "re-adopts a handed-off run that an older release already marked ownership-unverified": the stuck staging shape. Master: nothing claims it. - Related suites pass: `heartbeat-native-restart-suspension` (all 109), `services/heartbeat/recovery.test.ts` (all 103). - `tsc --noEmit` for `server`. ## Risks - A run that another live server prepares, but never claims, now stays `running` for up to 5 minutes before the existing rules apply. That is the same bound #21046 uses. - The guard trusts the shared owner prover. It reads "alive" only from a live controller instance lease or a live PID in the same PID space. - More handed-off runs reach the restart claim earlier. The claim keeps all its fencing and evidence checks; this change only makes the serving server ask the same question a new boot asks. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (code search, shell, test execution, read-only API inspection) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
a65644d1e1 |
fix(retry): continue an accepted connected Retry from recovered saved work when its chat replay is refused later (#21080)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Tasks from a connected chat (for example GitHub pull request reviews) show Retry when a run fails > - A person's Retry on such a task first asks the chat retry owner to repeat the exact failed chat request. #21038 made a refusal at the click continue the task from saved work instead > - The chat owner can also defer its source check (a busy identity, endpoint, or agent row lock) and refuse it a few seconds later. Then the route has already answered 202, and the task Retry receipt was marked failed with no run > - The task stayed blocked, and its execution blocker told the person to send a message, while the interruption projection still offered Retry > - This pull request makes that late refusal take the same board continuation as a refusal at the click, and makes the blocker offer Retry for recovered saved work > - The benefit is that an accepted Retry always starts work, and the two projections agree ## Linked Issues or Issue Description No public issue exists. This is the bug description, in the fields of `.github/ISSUE_TEMPLATE/bug_report.yml`. **What happened?** On a hosted deploy, about 20 blocked tasks of one busy GitHub review agent (`paperclip_runner`, native remote runs) did not restart after Retry. `POST /api/issues/:id/retry` returned 202 with `accepted: true`, a `chatActionId`, and `presentation.state: retry_scheduled`. No run was created. The task Retry receipt changed to `failed` 5 to 10 seconds later, and the task stayed blocked. The activity record of each click (`chat.failed_run_retry_requested`) has `sourceAuthorizationPending: true`, so the chat owner deferred its source check at the click. No chat wake row was written, so the chat replay ended before admission. The last run failed with `native_continuation_requires_reconciliation` / `runner_remote_process_identity_unavailable`. Its saved work was recovered (`workspaceRecovery.status: recovered`, remote lease destroyed with a termination receipt). The projections disagreed: `interruption` had `canRetry: true`, but `executionBlocker` had `canRetry: false` and `nextAction: "Saved work recovered. Send a message to continue."`. Root cause: when the chat retry action settles `failed`, `releaseUnadmittedTaskRetryInput` in `server/src/services/chat-channels/task-retry-input.ts` marked the linked task Retry receipt `failed`. #21038 added a board-continuation fallback only for a refusal inside the click transaction. A refusal after a deferred source check (for example 8 lock-busy attempts on the agent row, about 7 seconds) had no fallback. Separately, `getExecutionBlocker` hid Retry for every chat-bound task (`canRetry && !chatBinding`), and `workspaceRecoveryDisplay` always said "Send a message to continue" for recovered work. **Expected behavior** A Retry that is accepted starts work. If the exact chat replay is refused after the click, the person's Retry continues the task from its recovered saved work. The execution blocker and the interruption both offer Retry, and neither tells the person to send a message to unblock a failed run. **Steps to reproduce** 1. Bind a task to a GitHub chat conversation. Make its latest native run fail with `native_continuation_requires_reconciliation` and recover its saved work. 2. Hold the chat identity advisory lock while you click Retry, so the source check is deferred (`sourceAuthorizationPending: true`). 3. Hold a row lock on the agent while the chat retry processes, so the deferred check fails 8 times. 4. The route returned 202. The chat action settles `failed`, the task Retry receipt changes to `failed`, and no run starts. **Paperclip version or commit** `master` at |
||
|
|
cc6728e8de |
fix(recovery): continue upgrade-stopped Retry runs that the sweep refused or skipped (#21108)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Hosted servers are upgraded often. Each upgrade stops the old server gracefully, and a run that is still preparing ends `interrupted` (`server_shutdown_interrupted`) with bootstrap evidence that provider work did not start. > - #21078 let a run that a person started with Retry continue automatically after such a stop. After it deployed, a read-only audit of a staging server found four groups of Retry-started tasks that still did not continue: some were refused on every sweep, some were never looked at, and one Retry that a person pressed failed and was not retried. > - Product rule: an upgrade never leaves a task that needs a manual Retry, and a Retry a person presses must work. A refusal that cannot succeed later must not repeat forever. > - This pull request fixes each cause found, keeps every authorization check of the continuation gate, and ends the refusal loop with one plain escalation. > - The benefit is that Retry-started tasks continue through upgrades, including tasks earlier releases left waiting, with no manual step. ## Linked Issues or Issue Description Refs #21078, #21046, #21019, #14987, #19384 **What happened?** On a hosted staging server after #21078 deployed (read-only audit, run events and task activity): 1. A Retry-started task was refused every ~2 minutes ("The automatic retry could not revalidate the original user continuation."). While preparing, its Retry had switched to a fresh isolated workspace because the saved files could not be restored (activity `issue.fresh_workspace_selected` by the system, 40 minutes after the Retry). That rewrote the Retry request's strategy from `continue` to `fresh_isolated_workspace`, and `readTaskRetryRetryAuthority` accepts only `continue`. 2. Two tasks whose Retry replaced a native session had no recovery event for hours. The stranded-task sweep skipped them silently. 3. A task whose Retry selected a fresh workspace at admission (`explicitUserContinuation.readiness`) was refused on purpose ("fresh-workspace Retries keep their own contracts"), yet that contract had no path for a shutdown before provider work. The task stayed in progress after every upgrade. 4. A person pressed Retry. The run failed before provider work with `reusable_sandbox_resume_failed`, a transient failure that is already in the bounded bootstrap retry lane. One second later the stranded-task sweep escalated the task to `blocked`, which cleared the claim. Two seconds after that, the run's own failure path scheduled its retry and was refused because the claim no longer named the run. **Expected behavior:** each of these continues automatically once, under the person's original Retry. A refusal that cannot succeed later escalates the task once with a working Retry. **Root causes:** - Cases 1 and 3: the continuation gate refused every fresh-workspace Retry, whether the fresh workspace was chosen at admission or later in preparation. That includes one that a shutdown stopped before it did anything. - Case 2: the sweep's generic skips run before the continuation. Two of them can match only because of state that the replaced session left. An unfinished session goal (`agent_task_sessions.goal_status`) is skipped as "goal recovery owns it". A native finalization row is counted as a live execution path by `hasActiveExecutionPath`. The person's Retry had already started a fresh session, so neither owns the task. The audit could not read the database, so which of the two applied on staging is not proven; the tests cover both. - Case 4: a race. The sweep does not wait while a terminal run's own executor is still settling it, so it escalates before that executor schedules the run's bounded retry. - The refusal loop: after a refusal the sweep left the task `in_progress` and tried again on the next pass. **Paperclip version or commit:** master `482063aeca`. ## What Changed - `explicit-native-continuation.ts`: `readTaskRetryRetryAuthority` renews a fresh-workspace Retry **only** when the parent run was stopped by a graceful shutdown before provider work (`isShutdownInterruptedPreparation`). The admitted `readiness` must name the same Retry and source. A strategy that preparation switched to fresh is rebuilt exactly as preparation records it: this Retry, its source, and the source's leases. An automatic retry of the fresh attempt inherits the selection from its own receipt. The successor's receipt records the selection under `automaticRetry.readiness`. The successor's `explicitUserContinuation` keeps its plain shape. - `fresh-workspace-continuation.ts`: `readFreshWorkspaceReadiness` reads that receipt for an automatic continuation, so the successor prepares the same fresh attempt. A remote fresh workspace is keyed by the Retry, so it is reused rather than created again. `freshReadinessSchema` is exported and reused. - `recovery/service.ts`: - **Shutdown-stopped Retry preparations go first:** at the start of the stranded sweep, every `in_progress` task whose latest run is a legacy `server_shutdown_interrupted` preparation (`providerWorkStarted: false`) carrying `explicitUserContinuation` is offered to the scheduler. It is skipped when the agent is not invokable, the task has a live execution path, a board-owned recovery action or a pause hold, or the run is still settling. The unfinished-goal skip does not apply here, because the person's Retry started a fresh session. - `hasActiveExecutionPath` ignores a native finalization that a person's Retry already replaced (`failureDetail.replacementDenied = explicit_user_continuation` on a terminal run). - The sweep skips a task while its latest terminal run is still being settled: a live execution in this process, or another server's unexpired controller lease. It checks again when that lease expires. - A refused continuation (`continuation_user_authorization_missing`) escalates the task once to `blocked` with the plain notice "Retry needed … Press Retry to continue." Before, the task stayed in progress and was refused again on the next pass. - `heartbeat.ts`: `scheduleRecoveryRetry` reports a refused continuation to the sweep. - `doc/execution-semantics.md`: describes these rules. **Failure classes admitted for a Retry-started run (case 4):** no new class. The run's own bounded retry lanes are unchanged. `reusable_sandbox_resume_failed`, `remote_workspace_restore_failed`, `plugin_worker_unavailable`, `environment_provider_unavailable`, `repository_catalog_unavailable`, runner revision mismatch and the other transient bootstrap failures were already retried before provider work. The fix removes the race that took the claim away from that retry. **Refusals that remain by design:** - The task is reassigned, `done` or `cancelled`. - Another run holds the claim, or newer or live work exists. - The Retry actor, generation or run changed, or it is a chat or queued-message Retry. - The message was edited or deleted. - The receipt is missing or revoked. - The run had another interruption cause, or provider work started. - A fresh-workspace Retry whose run *failed* (not a shutdown) keeps its one-run contract. - A readiness that names another Retry or another source. Each of these now escalates the task once instead of refusing on every sweep. ## Verification - `server/src/services/task-retry.test.ts`, new tests. Red on `origin/master` sources and green on this branch (checked by restoring the master sources): - "continues a fresh-workspace Retry that a shutdown stopped before provider work": - Runs twice: once with the fresh workspace selected at admission, once with it selected at preparation. - Two concurrent sweeps create one successor, which gets the claim, and dispatch accepts it. - `readFreshWorkspaceReadiness(successor)` returns the same fresh attempt. - A second upgrade stops the successor, and the next successor carries the same attempt. - "continues a shutdown-stopped Retry whose replaced session left state behind": two cases, an unfinished session goal and a native finalization that the Retry replaced. - "escalates a refused shutdown continuation once instead of refusing it on every sweep": two sweeps. The task ends `blocked` with one "Press Retry to continue" notice and one refusal event. - "leaves a failed Retry run to its own executor while that executor still settles it": - Production shape: `reusable_sandbox_resume_failed` and a live controller lease from the server that ran the run. - The sweep leaves the task and its claim alone. - The run's bounded retry is then scheduled and dispatch accepts it. - "keeps a fresh-workspace Retry's own contract when it does not match": three refusal cases (other Retry, other source, failed rather than stopped). Green on both. - Existing suites pass: - `task-retry`, `explicit-native-continuation`, `heartbeat-retry-scheduling`, `heartbeat-process-recovery`, `interrupted-preparation-hold-settle`, `heartbeat-issue-liveness-escalation`, `issue-recovery-actions`, `recovery-deadline-acceptance`, `legacy-continuation-authority`, `workspace-recovery-admission`, `native-workspace-export-retry`, `heartbeat-paused-assignee-new-task`, `issue-tree-pause-resume-run`, `heartbeat-stale-queue-invalidation`. 1,619 tests passed, with one failure that a change in this branch fixed; rerun of the affected suites: 560/560. - The one failure came from the first version of the settling check, which also waited for this process's own lease after a shutdown drain. The check now uses this process's live executions and other processes' leases only. - `tsc --noEmit` for `server`. - Required CI covers the full suite. **After deploy, the next audit should show:** - Tasks stopped by a shutdown while their Retry was preparing get one `scheduled_retry` successor on the first sweep. This includes tasks whose Retry selected a fresh workspace and tasks whose Retry replaced a native session. - They show no further "could not revalidate" events. - A Retry-started run that fails transiently before provider work gets its bounded retry, and the task is not escalated in between. - A task whose continuation is truly refused is `blocked` once with "Retry needed". ## Risks - **Behavior change:** the sweep now offers shutdown-stopped Retry or message preparations to the scheduler before its generic skips. All authorization checks still run at admission, under the task and run row locks. The scheduler reuses an existing successor, so two servers create one continuation. The tests run two sweeps at once. - **Settling wait:** a task whose latest terminal run still has another server's live controller lease waits until that lease expires (at most 90 s) before the sweep acts. Normal turn starts are unchanged, and nothing is added before a turn starts. - **Escalation on refusal:** a refused continuation now moves the task to `blocked` with a Retry notice, where before it stayed `in_progress` with no path. Pressing Retry is a new, separately admitted request. - **Rows from older releases:** no schema change and no backfill. A shutdown-stopped fresh-workspace Retry is continued once. An older successor receipt without `automaticRetry.readiness` behaves as before. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (code search, shell, read-only API reads, Vitest on embedded PostgreSQL) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
1d153f62b6 |
feat(docker): build the image in the merge queue and promote it on master (#21096)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - `docker.yml` builds, probes, gates and signs the standard Core image for every master push. Paperclip Cloud deploys a Core commit only after that image is published. > - Each master push rebuilds the image from scratch: about 10.6 min for amd64 and 7.8 min for arm64. > - The owner approved a merge queue for Core. A merge queue tests the exact commit that master will become, before the merge. > - This pull request builds and probes the image for that exact commit in the queue, with no registry credentials. The later master push pushes those exact bytes instead of rebuilding. When no matching verified build exists, master builds from source exactly as today. > - The benefit is that the image legs on master drop from about 8–11 min to about 3–4 min, and nothing changes until the owner turns the queue on. ## Linked Issues or Issue Description Refs #21013 (it measured the image path and proposed this plan). **What existing behavior does this improve?** Every master push rebuilds the standard image, even when the same commit could already have been built while it waited to merge. **Subsystem affected** `.github/workflows/docker.yml` (`build-and-push` only), new `.github/workflows/docker-merge-group.yml`, new `scripts/merge-group-image.mjs`. **Current behavior** `build-and-push` always runs the full BuildKit build on master: 639 s (amd64) and 468 s (arm64) in run 43133035209. **Proposed behavior** On a `merge_group` event, build both architectures for the merge-group commit, probe PID 1, and upload an OCI archive plus a record of commit, tree, architecture and digest. On the master push for the same commit, each leg pushes that archive by digest instead of building. Everything after the leg (probe, exact-source gate, signed-index check, attestation) is unchanged. **Reason and benefit** The build moves before the merge, where it runs in parallel with the queue's other checks. The master leg only downloads and pushes. **Breaking changes** None. Without `merge_group` events, `find` returns no run and the leg builds as before. **Additional context** This PR does not touch the exact-source gate jobs (`source_verification`, `publication_verification`, `merge-and-push`), `scripts/watch-release-source.mjs` or `cloud-readiness.yml`. Another change owns them. ## What Changed - **`docker-merge-group.yml` (new, `on: merge_group` only).** It runs only for this repository's master queue. Workflow permissions are `{}` and the job has `contents: read` only. There are no secrets, no registry login, no `cache-to`, no `push: true`, no environments, and checkout uses `persist-credentials: false`. It runs on GitHub-hosted runners, not the paid AWS fleets. Steps: build `target: production` with the same build args as master (`PAPERCLIP_BUILD_COMMIT` is the merge-group SHA, which becomes the master SHA). Read the public registry cache. Write `type=oci` to a tar. `docker load` it and run `scripts/assert-orphan-reaping.sh`. Only then write `record.json` and upload `merge-group-image-<arch>` (3-day retention). - **`scripts/merge-group-image.mjs` (new).** - `find` (master push only): selects a completed, successful run of this workflow, with event `merge_group`, the same `head_sha`, repository and head repository both `1170821064`, and a `gh-readonly-queue/master/` head branch. It needs exactly one unexpired artifact for the architecture. - `record` (merge group only): reads the archive's `index.json`, checks each blob's sha256 and that the image is `linux/<arch>`, and writes `{sha, tree, arch, digest}`. - `verify` (master push only): re-reads the archive the same way and requires record SHA = `GITHUB_SHA`, record tree = `git rev-parse HEAD^{tree}` of the master checkout, same architecture, and archive digest = recorded digest. - **`docker.yml` `build-and-push`.** - The leg gets `actions: read` so it can download the artifact. - New steps run after the registry login: find, download, then promote (`verify`, then `skopeo copy --all --preserve-digests oci-archive:… docker://ghcr.io/…@<digest>`). The step sets its `digest` output only after the push succeeds. - All three new steps are `continue-on-error`. Buildx, labels, metadata and "Build and push by digest" run when `steps.promote.outputs.digest == ''`, so any failure builds from source. - "Select image digest" feeds the existing PID 1 probe and digest export. - **Tests (`scripts/merge-group-image.test.mjs`, 8 tests):** - Run selection rejects another commit, another event, another workflow, a fork head, another repository, another queue base, and failed, cancelled or running builds. - Artifact selection. - Archive digest with and without provenance, a tampered blob, the wrong architecture, and a real tar read through the system `tar`. - Record checks for commit, tree, architecture and digest mismatch. - The CLI refuses `record` outside `merge_group` and `find`/`verify` outside a canonical master push. - Static checks of the merge-group workflow: credentials, triggers, permissions and runners. - Static checks of the `docker.yml` wiring: promotion is gated and falls back, the output is set only after the push, and every build step has the fallback condition. ### How promotion proves it publishes the verified tree 1. `find` accepts only a run whose `head_sha` equals `github.sha` of the push, meaning the queue landed the commit it tested (unverified: see "Merge method" in Risks). 2. `verify` compares the record's tree with the tree of the master checkout, and its commit with `GITHUB_SHA`. 3. The archive is content-addressed. The digest is recomputed from the archive blobs and must equal the recorded digest. `skopeo --preserve-digests` to that digest reference fails rather than produce other bytes. 4. The leg then runs the PID 1 probe on that registry digest. `merge-and-push` still signs only an index whose Linux images are the probed digests (#21013), after the unchanged exact-source gate. 5. The pipeline itself is trusted only as master already ran it. `find` and `verify` run the copy of `scripts/merge-group-image.mjs` from the push's `before` commit (`git show "$BEFORE:…"`), not the copy the push brings. `verify` refuses promotion if anything in `before..HEAD` (every commit of a multi-PR group, not only the tip) changes `docker-merge-group.yml`, `docker.yml`, the promotion script or the PID 1 probe. Such a push builds from source. So does the first push after this PR merges, because its `before` has no verifier. ## Verification - `node --test scripts/merge-group-image.test.mjs`: 10 pass. One builds a real git history in which a two-PR push changes the pipeline in its first commit only, and checks that the range check catches it. - `node --test` on the workflow and contract suites (cloud-readiness, docker-canary-promotion, post-merge-runner-routing, lockfile-refresh-workflows, release-verify-workflow, reuse-canary-image, standard-image-contract, cloud-source-verification, merge-group-image): 492 pass. - `vitest run server/src/__tests__/docker-build-stamp.test.ts server/src/__tests__/container-init-reaping.test.ts`: 22 pass. - `actionlint` on both workflows: no new findings (the 6 remaining are pre-existing `ls` and word-splitting notes in `docker.yml`). - Local: `skopeo 1.24.1 copy --all --preserve-digests` between OCI layouts kept the index digest. It accepted a `docker://…@sha256:` destination and failed only at the network step. Cannot be exercised before the queue exists, because no `merge_group` event fires: - the merge-group run itself: the hosted-runner build time, disk use with tar plus `docker load`, `docker load` of a BuildKit OCI tar, and a ~5 GB artifact upload; - the cross-run artifact download on master; - whether `skopeo` exists on the runs-on AMI and accepts digest destinations in its installed version. Each master-side failure falls back to the full build. A merge-group failure only blocks the queue if its check is required, which is why that check is required last in the steps below. ## Risks - **Inert until enabled.** No `merge_group` events fire before the owner adds a merge queue. On master, `find` lists runs (one API call per leg, about 1 s), finds none, and the leg builds as today. The only always-on change is the leg's `actions: read` permission. - **Build cache freshness.** A promoted master push writes no `buildcache-<arch>`, and merge-group builds read but never write it. With every push promoted, the cache ages, so dependency or weekly tool-layer changes rebuild in each merge-group run until a fallback build refreshes the cache. A follow-up can refresh the cache from master without blocking publication. - **Publication time is still gate-bound.** The exact-source gate (about 13–14 min after push) still decides when the image publishes. See "Expected timing". - **Merge method (unverified).** Promotion needs the master commit to be exactly the commit the queue tested, because the image stamps the commit SHA (`PAPERCLIP_BUILD_COMMIT`, `build-info.json`, the provider pack revision). Matching only the tree would publish an image stamped with another SHA, so this PR does not relax the check. I expect the queue to fast-forward master to its tested commit. GitHub's merge-queue page does not state this, and it cannot be tested before the queue exists. If the queue lands a different SHA, promotion never matches and every push builds from source. That is safe but inert. Step 4 below checks this. - **skopeo on the AWS fleet.** The promote step checks for `skopeo` first and logs "Merge-queue promotion unavailable" if it is missing, then builds from source. If that notice appears, add skopeo to the fleet image. - **Hosted runner capacity.** amd64 merge-group builds run on 4-core hosted runners, colder and slower than the AWS fleet. That adds queue latency, not master latency. ### Settings the owner must enable, in order 1. **Prerequisite, a code change, not a setting.** The `master` ruleset (id 13619726) requires `ci / verify` and `ci / e2e`. Those come from `pr.yml`, which triggers only on `pull_request`. A merge queue waits for required checks on the `merge_group` event too, so every entry would stall until its timeout. First decide, and merge in a separate PR, how `ci / verify` and `ci / e2e` report on `merge_group`: a full CI run on the merge-group commit, or a fast pass that relies on the PR-head result. Today the ruleset does not require up-to-date branches either. 2. Merge this PR. Nothing changes yet. 3. In Settings → Rules → Rulesets → `master` (13619726), add **Require merge queue**: - merge method Squash; - build concurrency 5, minimum group size 1, maximum group size 5, wait 0 min; - "Require all queue entries to pass required checks"; - status check timeout 60 min. Do not add the image build to the required checks yet. Promotion is then opportunistic: if the merge-group build has not finished when master is pushed, master builds from source. 4. After the first queued merge, confirm promotion can match. The master commit SHA must equal the `head_sha` of its `Docker merge-queue build` run, and the next master `Docker` run's "Promote the merge-queue image" step must succeed. If the SHAs differ, promotion stays inert under this merge method. Stop here and report it. 5. After about a week of green `Docker merge-queue build` runs, add `build (linux/amd64, ubuntu-latest, amd64)` and `build (linux/arm64, ubuntu-24.04-arm, arm64)` (app: GitHub Actions) to the ruleset's required status checks. Then every queued commit has a verified image ready to promote. 6. The existing bypass actors (org admins, repository admins, integration 5534676) can still merge outside the queue. Those pushes have no merge-group run and build from source. ### Rollback - Remove the image checks from the required status checks (step 5), or remove **Require merge queue** (step 3). Master pushes then find no merge-group run and build from source. Nothing to migrate. - Code: revert this PR. Old merge-group artifacts expire after 3 days. Promoted images are ordinary digests under `ghcr.io/paperclipai/paperclip` and stay valid. ### Expected timing - Image leg on master: about 3–4 min (download about 5 GB, push the new layers by digest, PID 1 probe) instead of 10.6 min (amd64) and 7.8 min (arm64). - Push to published: still bounded by the exact-source gate on master, about 14–15 min (gate plus about 45 s for `merge-and-push`). Today it is the same bound, but with the legs as a second 11-min path and the serialized-run queue in front. - If the gate later accepts proof produced in the merge queue (owned by the gate change), push to published would be about 4–5 min (leg promotion plus `merge-and-push`). ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, read-only GitHub API and registry reads). Default reasoning mode. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
f8e1fb0278 |
perf(docker): take the orphan-reaping pull and pack chmod off the image path (#21013)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - `docker.yml` builds and signs the standard Core image for every master push. Paperclip Cloud deploys a Core commit only after this image is published and attested. > - A master commit takes 29 min (p50) and 56 min (p90) to reach a published image. Every minute here delays a staging fix. > - The run logs show where the time goes. The exact-source gate (about 14 min) is the critical path today. The build legs finish about 3 min before it. After the gate, `merge-and-push` spends about 95 s to pull the 4.9 GB amd64 image for one probe. > - This pull request moves that probe into the build legs, which run before the gate. It also removes a duplicate 187 MB layer from the amd64 image. > - The benefit is about 95–100 s less from gate to signed image on every master commit, with the same proof. arm64 is now also probed. ## Linked Issues or Issue Description **What existing behavior does this improve?** The time from a commit landing on master to the signed standard image (`ghcr.io/paperclipai/paperclip:sha-<full>` plus its attestation). **Subsystem affected** `.github/workflows/docker.yml`, `Dockerfile`, `scripts/standard-image-contract.mjs`. **Current behavior** Measured on run 43133035209 (sha |
||
|
|
533f135780 |
fix(native): send a re-wake whose crashed predecessor's warm Runner cannot be adopted to a fresh sandbox instead of failing it (#21001)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native run on a warm remote sandbox leaves its Runner `ready` in the sandbox. The next run of the task adopts that Runner live, after it proves that the previous owner finished > - When the server is killed after a run parked on a question, the warm owner lived only in the killed server's memory. The Runner in the sandbox was never suspended > - A re-wake on the replacement server resumed that sandbox and tried live adoption. A proof failed, and the new run failed with `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate` and needed a Retry, although it did no provider work > - #20863 already sends such a run to a fresh sandbox when the predecessor's Runner cannot be retired, but the live-adoption path of a resumed sandbox (`remote-warm-session-recovery.ts`) was not connected to it, and the failed run's inherited Runner identity blocked the "workspace unchanged" proof that the fallback needs > - This pull request connects the two: a refusal of live adoption after the predecessor is durably terminal with an accepted result goes to the #20863 fresh-sandbox path, and an inherited Runner identity no longer counts as a process of the follow-up > - The benefit is that a re-wake seconds after a crash restart continues without a Retry ## Linked Issues or Issue Description Refs #20863, #20844, #20928, #20979, #20953, #20972 **What happened?** Restart-chaos wave `w-20261112T012841-s7000` (master `4b24352712`, volume-free mode, Daytona warm, Codex, 10 tasks, K1 and K15 with a fresh home and overlap), task t08 (W2Q): - Run `58f57bf1` asked its question and succeeded at 01:29:56. Its warm owner stayed in the server's memory. - K1 killed the server at 01:30:06. The replacement was ready at 01:30:15. - A board comment woke the task at 01:30:17. Run `8f812df6` started at 01:30:19 and resumed the same sandbox (`run.startup.reuse` `verified_reuse`). - The first live-adoption inspection passed. The second check (`revalidate`, in backend setup, before the Runner attach) refused at 01:30:21 (stack: `remote-warm-session-recovery.ts:217` from `revalidate`). The run failed `native_session_interrupted` with `INSTRUCTION_COLLECTION_UNAVAILABLE`. - The failed run kept its full save because the unchanged-files proof refused with `process_recorded`: the run carried its predecessor's Runner process identity. Retry (`6968dc0f`) worked. **Expected behavior** The predecessor finished with an accepted result before the kill, and the new run did no provider work. The task must continue without a Retry. **Steps to reproduce** 1. Finish a warm remote run that parks on a question. 2. Kill the server (SIGKILL) and start a replacement. 3. Wake the task within seconds. The new run resumes the warm sandbox. 4. Any live-adoption proof fails, and the run fails terminally with `prior_owner_terminal_state_indeterminate`. **Which proof failed** The old refusal had no reason. The wave evidence proves that it was `revalidate`, after a passed inspection, and not the SIGTERM of the next K15 fault (that server logged "shutdown signal received" only at 01:31:09). `revalidate` checks the predecessor's coordinator row, the current run's coordinator row and lease, and the task. The evidence cannot tell which one changed in the first seconds after the restart. This pull request names the failed proof (`warm_recovery_refused=<reason>`), so the next occurrence identifies it. The fix covers the class: every refusal that concerns the predecessor or the sandbox. ## What Changed - `server/src/services/native-runtime/remote-warm-session-recovery.ts`: - Each refusal names its proof from a closed list: `identity_incomplete`, `target_not_resumed_sandbox`, `owner_unsettled`, `execution_mismatch`, `checkpoint_unsettled`, `journal_unsettled`, `controller_not_stopped`, `prior_lease_mismatch`, `owner_changed`, `current_lease_changed`, `current_run_not_admitted`, `current_owner_changed`. - After the predecessor's run is `succeeded` and its coordinator is `committed` with a result and no lease, a refusal about the predecessor, its journal or the sandbox throws `NativePredecessorRetirementUnprovenError` (predecessor run, sandbox). The existing #20863 path then marks the sandbox `replaceBeforeReuse` and re-queues the run in the uncharged `interrupted_preparation` lane. The fresh sandbox is used only when the workspace save is clean and the harness backup is verified (unchanged #20863 safety rules); otherwise the sandbox is resumed as before. - A refusal because the new run is no longer admitted (stopped, reassigned, paused, company inactive) or no longer owned by this controller stays an ordinary failure, so a fallback cannot become a path around Stop or reassignment. - `server/src/services/remote-workspace-unchanged-proof.ts`: a run whose recorded Runner process identity is exactly a settled predecessor's identity of the same session (`carriesInheritedRunnerIdentity`, from #20979), with no `native.process_identity_recorded` event of its own, is no longer refused with `process_recorded`. A provider session of its own (`sessionIdAfter`) and every other check still refuse. - `doc/execution-semantics.md`: document both rules. ## Verification - Red on `origin/master` (`8818402e7a`): - `cd server && npx vitest run src/services/native-runtime/native-controller-recovery.integration.test.ts -t "crash restart"`: `AssertionError: expected Error: runner_state_identity_mismatch: pr… to be an instance of NativePredecessorRetirementUnprovenError`. - `npx vitest run src/services/remote-workspace-unchanged-release.test.ts -t "follow-up that carries"`: `expected false to be true` for the inherited-identity case; the own-identity and own-record cases passed (still refused). - Green with the fix: `native-controller-recovery.integration`, `remote-workspace-unchanged-release`, `native-recovered-owner-followup` and `heartbeat-native-predispatch-retry` pass (149 tests). Also run: `native-session-executor.test`, `explicit-native-continuation.test`, `task-workspace-binding.test`, `native-runner-restart-recovery.integration.test` (see the PR comment for the result). `tsc --noEmit` for the server passes. - The new integration case is the staging shape: a predecessor that parked on a question (unanswered `semantic_tool.input` before its terminal), a replacement controller, a resumed sandbox. It checks the journal refusal, a predecessor row taken by another worker between inspection and revalidation, a changed current lease, and that a cancelled current run is not sent to the fallback. - Chaos: the before/after rerun of seed 7000 is in a PR comment. ## Risks - The fallback starts the next attempt on a fresh sandbox instead of reusing the warm Runner. That costs one cold start, only after live adoption is refused. - The unchanged-files proof accepts an inherited Runner identity. The identity must match a settled predecessor of the same session exactly (PID, group and start time), and the run must have no own identity record, no execution event, no runner checkpoint, no workspace operation and no result. These are the same facts #20979 uses for restart recovery. - Error text of this refusal gains `; warm_recovery_refused=<reason>`. Callers match the `runner_state_identity_mismatch: ` prefix only. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
c4927e1ec2 |
fix(recovery): continue a Retry-started run that a server shutdown stopped while preparing (#21078)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - An upgrade stops the old server gracefully. A legacy run that is still preparing ends `interrupted` with `server_shutdown_interrupted`. The new server must continue it automatically (#21019, #21046). > - A run that a person started with Retry (or a message) carries `explicitUserContinuation`. Its automatic retries need their own receipt from `admitExplicitContinuationRetry` (#14987, #19384). > - That admission refused every shutdown-interrupted Retry run. The task then stayed in progress with no run, and the stranded-task sweep refused it again about every 2.5 minutes, forever. A person had to press Retry. > - An upgrade is routine. It must never leave a task that needs a manual Retry. > - This pull request lets the person's original authorization cover a continuation after a shutdown that stopped the run before provider work started. All other checks of the gate stay. > - The benefit is that Retry-started tasks continue through an upgrade, and tasks that earlier releases left waiting continue on their own after deploy. ## Linked Issues or Issue Description Refs #21019, #21046, #14987, #19384 **What happened?** On a hosted staging server, each automatic upgrade stopped legacy runs that were still preparing (`executionRecovery {kind: bootstrap, providerWorkStarted: false}`). When the run came from a person's Retry (wake reason `task_retry`), the continuation was refused with "The automatic retry could not revalidate the original user continuation." (`continuation_user_authorization_missing`). The task stayed `in_progress` with no run and no next retry. Runs from older upgrades were refused again by each sweep. A run from the same upgrade that did not come from a Retry got its continuation in the `interrupted_preparation` lane. **Expected behavior** A Retry-started run that a graceful shutdown stopped before provider work continues automatically, once, under the person's original Retry. Tasks that an earlier release already refused continue after deploy with no manual step. **Root cause** Three conditions refused it (`server/src/services/heartbeat/retries.ts`, `server/src/services/explicit-native-continuation.ts`): 1. The scheduler admits an explicit continuation only for the `transient_failure`, `server_restart`, preparation-wait and agent-directory lanes. A shutdown-interrupted preparation with unspent failure budget uses `interrupted_preparation`, so it was refused before admission ran. 2. `admitExplicitContinuationRetry` accepted only a `failed` or `timed_out` parent. These parents are `interrupted`. 3. It required `task.executionRunId` to be the parent. The refusal released the claim (`releaseIssueExecutionAndPromote`), so every later sweep was refused by this check too. **Paperclip version or commit** master `f30005440b`. ## What Changed - `retries.ts`: an explicit continuation may also use the `interrupted_preparation` lane. Lane choice is unchanged (see below). - `explicit-native-continuation.ts`: `admitExplicitContinuationRetry` also accepts a parent that is `interrupted` by a graceful shutdown (`isServerReplacementInterruption`, so `server_shutdown_interrupted`) with bootstrap evidence `providerWorkStarted: false` (`isShutdownInterruptedPreparation`). - For such a parent only, a released claim (`executionRunId` null) is accepted. Admission restores the claim to the parent under the task lock, as the shutdown path leaves it, so the scheduler passes the claim to the successor and dispatch accepts it. If the parent validation then refuses, admission releases the claim again in the same transaction. A claim that another run holds still refuses. - `doc/execution-semantics.md`: describe the rule and the conditions that still refuse. Also apply the sentence that #19384 left pending (a plain task Retry is renewable). - No schema change, no backfill. Nothing changes on the path that starts a turn. **Which lane a graceful-shutdown legacy preparation uses (after #21046).** `scheduleBoundedRetryForRun` sets `interruptedPreparation` for an `interrupted` run with the bootstrap evidence. `usesServerRestartLane(run, true)` returns false while the failure budget is unspent, so the run uses `interrupted_preparation`: it does not spend the failure budget, the first continuation is due at once, and it is bounded by `INTERRUPTED_PREPARATION_RETRY_MAX_ATTEMPTS`. With a spent failure budget it uses `server_restart`. This is the intended design (#21046 comment on `usesServerRestartLane`), so no lane change. The `server_restart` lane is for `process_lost` replaced preparations and for runs past dispatch. **Conditions that still refuse** (each has a test or an existing test): - The task has another assignee, or is `done` or `cancelled`. - Another run holds the task's claim. - Newer work on the task (a run created after the parent, a retry child, or a live run). - The Retry request changed: other actor, not admitted, other generation, bound to another run, a chat Retry, a fresh-workspace Retry. A message continuation with an edited or deleted message. - A missing or revoked receipt (dispatch validation of the parent fails). - An execution blocker on the task, or a live adapter controller for the parent. - An `interrupted` parent with another cause (for example an operator interrupt), or with no bootstrap evidence, or with `providerWorkStarted: true`. A parent that failed after provider work keeps the existing rules (`failed`/`timed_out` only). - Native runtime parents (unchanged, see below). **Native runtime.** A native `paperclip_runner` run is not interrupted by a graceful shutdown: `drainRunningRunsForShutdown` suspends it and the next server reattaches it (`restart_suspended`). It does not reach this gate. Native replacement paths (`native-safe-replacement`, `native-provider-capacity-retry`) remove `explicitUserContinuation` from the successor and use their own contract. So the native path does not have this refusal, and this pull request does not change it. ## Verification - `server/src/services/task-retry.test.ts`, new tests. Red on `origin/master` (no successor), green on this branch: - "continues a Retry-admitted run that a graceful shutdown stopped while it was preparing": production shape. A legacy run admitted by a task Retry, `preparing`, owned by this server, with bootstrap evidence. `drainRunningRunsForShutdown("SIGTERM")` interrupts it with `server_shutdown_interrupted` and schedules one `interrupted_preparation` successor (`failureRetries: 0`, `interruptedPreparations: 1`) with its own receipt. The claim passes to the successor and dispatch accepts it. A concurrent stranded sweep and a second schedule call reuse the same successor. - "continues a task an earlier release left waiting after refusing its shutdown continuation" (a direct Retry run, and an automatic retry of a Retry run): the row and task as an older release left them (claim released). Two concurrent stranded sweeps schedule exactly one successor; the claim passes to it; dispatch accepts it. - "does not continue a shutdown-interrupted Retry run without its original authority" for: provider work started, no bootstrap proof, another interruption cause, claim held by another run, reassigned, superseded, other Retry actor, cancelled task. No successor, no receipt, and the claim is left as it was. - I confirmed that each source change is needed: without the lane change both main tests fail; without the released-claim rule the older-release test fails. - Suites: `task-retry`, `explicit-native-continuation`, `legacy-explicit-continuation-recovery`, `quarantined-task-retry` (587/587); `heartbeat-process-recovery`, `heartbeat-retry-scheduling` (520/520). - `tsc --noEmit` for `server`. - `pnpm verify` selects a broad run (exit 2) because `retries.ts` is central; required CI covers that. **After deploy, the next audit should show:** a Retry-started run that a shutdown stops while preparing gets a `scheduled_retry` successor with `scheduledRetryReason: interrupted_preparation` (or `server_restart` with a spent failure budget) and no `continuation_user_authorization_missing` event. The tasks that earlier upgrades left waiting get one such successor on the first stranded sweep of the new server, with the task claim on the successor. ## Risks - Behavior change: a shutdown-interrupted preparation of a Retry or message turn now continues automatically, within the existing uncharged and bounded lanes. The run did no provider work, so there is nothing to reconcile. - Rows saved by older releases: a refused row whose claim was released is continued once. If another run took the claim, or newer work exists, it stays refused. - Two servers at once: admission runs in the scheduler transaction under the task and run row locks, and the scheduler reuses an existing successor (`retryOfRunId`). The tests run two sweeps concurrently and get one successor. An old server that still refuses cannot clear the claim after the new server moved it to the successor (its release matches only the stopped run). - The claim restore writes `executionRunId` for a terminal run inside the scheduling transaction. If admission refuses after the restore, it releases the claim again before the transaction commits. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (code search, shell, Vitest on embedded PostgreSQL) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
f30005440b |
fix(recovery): reconnect a live remote Runner with its admitted skills instead of re-downloading them (#21064)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Agent turns run in a remote Runner sandbox. A server upgrade
detaches each running turn, and the new server must reconnect to its
Runner (restart recovery).
> - A reconnect runs the full turn preparation before it contacts the
Runner. On a new server, skill preparation downloads every saved skill
version again, under one lock for the whole company.
> - After an upgrade, many reconnects wait behind each other's
downloads. The Runner stops answering after about 2 minutes, so a late
reconnect loses the turn.
> - This pull request lets a reconnect to the same run's live remote
Runner use the exact saved skill versions, without downloading them and
without the lock.
> - The benefit is that skill preparation no longer adds to the
reconnect delay, and it does not grow with the number of reconnects.
Normal turn starts do not change.
## Linked Issues or Issue Description
**What happened?**
After an automatic staging upgrade, runs that the new server reached
late failed. The measured time from detach to reconnect was 114–152 s
for the failed runs. Turn preparation took 30–130 s, and skill
preparation alone took 46 s on one run.
**Expected behavior**
A reconnect to a live Runner reaches it quickly. Its speed must not
depend on how many other runs prepare skills at the same time.
**Steps to reproduce**
1. Run a remote Runner turn for each of many tasks in one company.
2. Upgrade the server while those turns run, so the new server starts
with an empty skill cache.
3. See that each reconnect waits in
`heartbeat.prepare.configuration_skills` for the company skill cache
lock and downloads the skills again.
**Paperclip version or commit**
master at
|
||
|
|
8e956da1cf |
fix(native): keep a live reattached Runner when its restart recovery read times out (#21053)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs on remote sandboxes (Daytona) use a detached Runner. A server replacement detaches the controller and leaves the Runner alive. The new server reattaches to it and continues the same turn > - On staging, about half of the in-flight runs at each upgrade lost their turn. The new controller reattached to the live Runner (same process verified), its recovery read timed out (`PRP command session.snapshot timed out`), and the failure cleanup killed the Runner > - The next same-run reattachment then found `process_not_running`. #21039 continues these runs in a fresh session, but the in-flight turn is lost and its outcome is unknown > - The cause of the kill is in the Codex driver's `recoverSession`: its cleanup closed the transport without retention. For a reattached Runner that means SIGKILL after the close deadline. #21023 already keeps the Runner for a failed open session, but not for a failed recovery > - This pull request keeps the live Runner when a recovery fails or is aborted on its way to it. Identity refusals and protocol integrity faults still stop it > - The benefit is that a slow reattachment no longer destroys a turn that was still running ## Linked Issues or Issue Description Refs #21023, #20923, #21039, #21052 **What happened?** Staging replaced its server at about 07:10Z with 17 remote native runs in flight. For each, the old server logged "Server shutdown suspended native controller ownership without cancelling provider work". The new server reattached. Example run: the Runner process (same pid and start time) was verified at 07:12:37 and the transport authenticated at 07:12:32. At 07:13:28 the recovery failed with `native_session_recovery_failed: ... PRP command session.snapshot timed out`, and a same-run resume was saved. At 07:14:12 the same-run reattachment found `process_verification` / `process_not_running`, with the Runner journal still `ready` and no exit record from the Runner itself. The run failed with `runner_remote_process_identity_unavailable` / `native_continuation_requires_reconciliation`. 90 of 93 such failures in two hours of upgrades had this sequence. **What did you expect to happen?** A recovery that times out against a live Runner is a transient failure. The same-run reattachment should find the Runner alive and adopt it again. Killing the Runner is only for a deliberate stop, an identity refusal, or an integrity fault. **Root cause** `packages/paperclip-runner/src/drivers/codex/codex-app-server-driver-impl.ts` `recoverSession`: the `catch` block and the abort handler call `bootstrapCancellation(...).close()`, which calls `transport.close()` without `retainUnsettledRemoteRunner`. In `runnerd-codex-transport.ts` `#closeOnce`, an unresponsive adopted Runner that is not retained gets `SIGKILL` after the close deadline. The Codex driver serves remote Codex, OpenCode and ACPX runs, which matches the failures seen on staging for all three driver kinds. ## What Changed - `codex-app-server-driver-impl.ts`: `recoverSession` classifies a thrown recovery failure. A transient one (`provider_transport_failed`, a `PRP command ... timed out`, or the transport closing while waiting) now detaches this controller with the transport's restart detach (`detachControllerForRestart`). That path queues no `turn.stop` and no `runner.suspend` and sends no signal, the same as a server restart. The Runner and its in-flight turn stay for the same run's next reattachment. An abort (a Stop or the recovery deadline), any message that names a mismatch, a different session, identity, authority, integrity or incompatibility, any unclassified failure, any returned identity refusal, and a protocol integrity fault still close and stop the Runner. `openSession` is unchanged. - `codex-app-server-driver.recovery.test.ts`: the staging message (`codex_history_read_failed: ... provider_transport_failed: PRP command session.snapshot timed out`), and a plain snapshot timeout now detach and never close. An abort of a hung read still closes and stops the Runner. A thrown different-session refusal, `native_adopted_provider_identity_mismatch`, an identity refusal that also mentions a timeout, and an unclassified failure still close without retention. The existing integrity-fault cases check that the close does not retain the Runner. - `doc/execution-semantics.md`: describes the rule. ## Verification - Red on `origin/master` with only the new tests: `cd packages/paperclip-runner && npx vitest run src/drivers/codex/codex-app-server-driver.recovery.test.ts` fails 2 of 69 (the two detach cases; master calls `close()` and stops the Runner). - Green with the fix: 69 of 69. - `npx vitest run src/drivers/codex src/native-session-runtime.test.ts src/backends src/live/runnerd-codex-transport-remote-retention.test.ts --maxWorkers=4` passed except two timing tests (Codex command resolution, background cleanup containment) that pass alone and do not touch this code. - `npx tsc --noEmit -p packages/paperclip-runner/tsconfig.json` passes. - `detachControllerForRestart` is the existing restart path. It stops the pump, disconnects the Runner, and drains the controller journal. It queues no command and never signals the process. - Not run: a live Daytona restart. The next post-upgrade audit should show fewer `runner_remote_process_identity_unavailable` failures after a `session.snapshot timed out` resume. Instead, the run log should show "controller authority detached for restart; durable runner left alive" and then a successful same-run reattachment. ## Risks - Medium. A Runner that the recovery could not reach now stays alive, with its turn, for the same-run reattachment. This is the state a server restart already leaves. The same-run reattachment adopts it, or a Stop cleanup or saved-work recovery stops it. - If the Runner stays unresponsive across every same-run attempt, the run no longer gets the `process_not_running` proof that #21039 uses for its automatic fresh-session continuation. The run then keeps its visible Retry. The staging data shows that reattachments reaching the Runner sooner succeed, so a later attempt is expected to succeed. This is not proven against a live sandbox. - An abort of the recovery (a Stop or the recovery deadline) is unchanged: it closes and stops the Runner. `plugin-runner-release-resume.integration.test.ts` (14 of 14) covers a Stop during an adopted resume. - No turn is replayed and no second Runner is launched. Adopting an existing Runner never launches a new one. - Previous-release story: no persisted state changes. The decision is made in the process that runs the recovery, so a deploy during a run changes nothing. - No wait is added before a normal turn start. Only the recovery failure path changes. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude (Anthropic) via Claude Code, with tool use (shell, file edits, test runs, read-only staging API reads). Model ID as attributed in the commit trailer: Claude Fable 5.2. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
424e2c6488 |
fix(recovery): keep live owners' runs through an upgrade and continue replaced preparations in the restart lane (#21046)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - An upgrade replaces the server. The old and the new server serve together for several minutes. A legacy run proves that its owner is alive by renewing a 60-second lease on its `heartbeat_runs` row. > - #21019 added an uncharged `server_restart` retry lane for runs that a restart or replacement ended. On the first upgrade that shipped it, no run used that lane. The new server also failed many runs as `process_lost` while the old server that owned them was still serving. > - Each of these runs then spent one failure retry. A run that was already the last automatic retry was held for manual reconciliation, so the task needed a person. > - An upgrade is routine. It must not cost a task its work or a retry, and it must not leave a task waiting for a manual Retry. > - This pull request sends a run that a replacement stopped during preparation to the `server_restart` lane. It also stops the reaper from failing a run while the server process that owns it is proven alive. > - The benefit is that a task continues by itself through an upgrade, also when the database is slow during the swap. ## Linked Issues or Issue Description Refs #21019, #21042 **What happened?** A hosted staging server was replaced by an automatic upgrade. The new server started at 06:18:46Z, became ready at 06:26:17Z, and the old server received SIGTERM at 06:27:34Z. Between 06:23Z and 06:26Z the new server failed 44 runs as `process_lost` ("Process lost -- server may have restarted; retrying once", `processLossDiagnostic.controllerReplaced: true`). The old server still owned these runs and was still serving. Every reaped run was a Runner run in `executionStage: preparing` with no process recorded. Each successor was scheduled with `scheduledRetryReason: transient_failure` and `executionRetryAccounting.failureRetries: 1`. No run on the server has `retryReason: server_restart` or `executionRetryAccounting.serverRestarts`. The run evidence also shows why the leases lapsed. The old server's lease renewals waited 19 to 50 seconds on the database (`controllerLeaseLoss.renewalPendingMs`) while its timers fired on time (`deadlineLateMs` 0 to 17 ms). #21042 fixes that renewal stall. This pull request fixes how the reaper and the retry scheduler treat these runs. **Expected behavior** - A server never fails a run as lost while the server process that owns it is alive and serving. - A run that a replacement really ended continues once, automatically, without spending the failure budget. **Steps to reproduce** 1. Seed a legacy `paperclip_runner` run in `executionStage: preparing`, owned by another server boot, with a lapsed run lease. 2. Run `reapOrphanedRuns()` on the new server. 3. See that the successor is `transient_failure` with `failureRetries: 1`. When the chain already spent its failure budget, see that no successor is created and the run is held for reconciliation. 4. Give the owner a live controller instance lease (the old server is still serving) and run `reapOrphanedRuns()` again: the run is failed as `process_lost` anyway. **Root cause** - `usesServerRestartLane` (`server/src/services/heartbeat/retries.ts`) left every run with the bootstrap evidence `providerWorkStarted: false` to the `interrupted_preparation` lane. But `scheduleBoundedRetryForRun` gives that lane only to an `interrupted` run (or a native preparation). The reaper writes the same evidence on a `failed`/`process_lost` run that it stops during preparation. That run fell through to the charged `transient_failure` lane. The #21019 tests used runs in `dispatching`, so they did not catch it. - `legacyExecutionNeedsReconciliation` held every run whose failure budget was spent before it checked the bootstrap evidence. A run that a replacement stopped before dispatch did no provider work, but it was held for reconciliation, and the task waited for a person. - The orphan reaper (`recoverActiveExecutions`, `server/src/services/heartbeat/recovery.ts`) treated a lapsed run lease as proof that the owner was gone. It checked the shared owner prover only for "gone", never for "alive". During the swap the old server's renewals were slow, so its leases lapsed while it was alive. **Paperclip version or commit** Staging Core `51d77648bc` (first build with #21019). The behavior is the same on master `148d2b629c`. ## What Changed - `retries.ts`: `usesServerRestartLane(run, preparationLaneTakesRun?)` keeps the `interrupted_preparation` exception only when that lane really takes the run. `scheduleBoundedRetryForRun` passes its own verdict. Without one, only an `interrupted` run qualifies. A `process_lost` run that a replacement stopped during preparation now continues in the `server_restart` lane. - `legacy-execution-recovery.ts`: a replaced preparation (`isReplacedPreparation`) does not need reconciliation, also when its failure budget is spent. The `server_restart` bound limits it. - `recovery.ts` (orphan reaper): - When a run's lease lapsed but the shared owner prover proves its owner process alive (for example, the owner's controller instance lease is live), the reaper leaves the run with that owner. The owner stops and settles the run itself (`controller_lease_lost`, uncharged `interrupted_preparation` lane). The reaper recovers the run once the owner is gone or stopped. It also recovers the run when the owner has not settled it within `LEGACY_LIVE_OWNER_SETTLE_GRACE_MS` (5 minutes) after the lease lapsed. - A setup that a replacement stopped before dispatch uses the `server_restart` bound for its retry decision, whatever the adapter. - `legacy-controller-lease.ts`: add `proveLegacyController(run)`, which returns the shared prover's verdict. `legacyControllerProvenGone` uses it, and the reaper computes the proof once per run. - `server-replacement-interruption.ts`: add `isReplacedPreparation(run)`: bootstrap evidence `providerWorkStarted: false`, a recorded controller process stamp, and a server-replacement interruption. A row saved by an older release without the stamp keeps the charged lane (the existing documented rule). - `legacy-execution-recovery.ts` (`settleInterruptedNativeBootstrap`): a reconciliation hold that an older release saved on a replaced preparation is settled once its owner is proven gone, even when the failure budget is spent. The run continues in the `server_restart` lane. - `legacy-controller-lease.ts`, `recovery/ownership-registry.ts`: the `legacy_interrupted_preparation_hold` kind judges a finished run's hold by its recorded controller process (`finishedLegacyControllerOwnerStamp`). Revocation by the orphan reaper swaps in a random boot id, so before this change a reaped preparation's hold had owner "none" and was never settled. - `recovery/service.ts`, `heartbeat.ts`: the stranded-task sweep reports exhaustion from the scheduler's own `retry_exhausted` result, so a spent `interrupted_preparation` lane blocks the task even when the synchronous budget check reads an unspent restart bound. - `doc/execution-semantics.md`: document these rules. - No schema change. Nothing changes on the path that starts a run or a turn (no new waits and no file work). ## Verification - Red on `origin/master`, green on this branch (`server/src/__tests__/heartbeat-process-recovery.test.ts`, block "runs ended by a server restart or replacement"): - "continues a first attempt / a run that was already the last automatic retry that a replaced server left preparing, in the restart lane". This is the production shape: a `paperclip_runner` run in `preparing`, two servers sweep at the same time. Master: `transient_failure` with `failureRetries: 1`, and no successor for the spent chain. Now: one `server_restart` continuation, due at once, with the failure count unchanged. - "leaves a preparing/dispatching run whose lease lapsed with its owner, then recovers it once the owner stops". The owner's controller instance lease is live and two servers sweep. Master: `process_lost` at once. Now: the run stays `running` with its owner. After the owner stops, it becomes `process_lost` and a `server_restart` continuation is scheduled. - "recovers the run when its live owner has not settled it within the grace". - `interrupted-preparation-hold-settle.test.ts`: "releases an older release's hold on a preparation a server replacement stopped with its failure budget spent" for an unrevoked and a reaper-revoked lease (red without the settlement and owner-stamp changes) and "blocks the task when a replaced preparation's lane is spent even though the restart bound is not" (red without the sweep change). - Related suites: heartbeat process recovery, retry scheduling, orphaned active lease sweep, recovery deadline (bootstrap and ordinary bootstrap), legacy continuation authority, undispatched execution wait, agent status recovery, legacy workspace restore recovery, legacy execution recovery, legacy controller lease, server replacement interruption, heartbeat recovery, provider failure classification, reconciliation run read, and wake-queue policy. - `tsc --noEmit` for `server`. ## Risks - A run whose owner is alive but no longer drives it now stays `running` for up to 5 minutes after its lease lapsed, and then the reaper recovers it. The owner's own sweep still recovers a run that the owner itself lost the handle for (this boot is never deferred). - The guard trusts the shared owner prover. That prover reads "alive" only from a live or grace controller instance lease, or from a live PID in the same PID space. A server whose instance lease cannot renew for 60 seconds stops itself, so a stuck owner cannot hold runs forever. - More runs continue in the uncharged lanes. The `server_restart` bound (6 in 60 minutes) still stops a loop, and the stop is shown on the task. - Rows saved by older releases: a `process_lost` row with the bootstrap evidence and an old-boot owner now continues in the `server_restart` lane instead of being held. These runs did no provider work. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (code search, shell, test execution, read-only log and API inspection) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
fd5ee7e385 |
fix(native): continue a routine assignment or chat message whose remote Runner an upgrade lost mid-turn (#21052)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs on remote sandboxes (Daytona) use a detached Runner. When the server is replaced during an upgrade, the new server reattaches to that Runner > - On staging, upgrades every 15–40 minutes lose the in-flight Runner. The same-run reattachment then finds `process_verification` / `process_not_running`, the run fails with `runner_remote_process_identity_unavailable`, and the task is blocked > - #21039 made these runs continue once in a fresh session (the turn is recorded as having an unknown outcome and is not replayed). It covers user wakes and automatic retries (`retryOfRunId`) only > - A routine's system assignment (`issue_assigned`, requested by the system) and an external chat message wake have no `retryOfRunId`. They got `not_user_request` and stayed blocked for a person > - This pull request treats those two wakes as owed task work, so they continue the same way once the Runner is proven gone > - The benefit is that an upgrade no longer leaves these tasks waiting for a Retry click ## Linked Issues or Issue Description Refs #21039, #21023, #20923 **What happened?** Staging replaced its server at about 07:10Z. 17 remote native runs were in flight. Each new controller reached its Runner 114–152 s after detach. Recovery failed with "PRP command session.snapshot timed out". The same-run reattachment then found the Runner process gone, and each run failed with `native_continuation_requires_reconciliation`. #21039 continued 14 of the 17 automatically. The other 3 were a routine's assignment (two runs) and an external chat message. They have no `retryOfRunId`, so `requestInterruptedNativeTurnRetry` returned `not_user_request`. Those tasks are still blocked with no retry. **What did you expect to happen?** The Runner is proven gone, so nothing can still run the turn. The run should continue once in a fresh session, the same as an automatic retry of owed work. **Steps to reproduce** 1. Start a remote native run from a routine assignment, or from an external chat message. 2. Replace the server while the provider turn is active, so that the reattachment finds `process_not_running`. 3. The run fails and the task stays blocked. No automatic Retry is requested. ## What Changed - `server/src/services/heartbeat/restart-interrupted-turn.ts`: after the Runner is proven gone, a system wake also continues when it is a system assignment (`issue_assigned`), or when it is a comment or external chat message wake whose message is still on the task (not deleted, not written by a run). It runs under the run's responsible user, at most once per chain of runs, like an automatic retry already does. A linked person's external chat message continues under that person when the message is still on the task. Monitors and other system wakes are unchanged. - The same file and `server/src/app.ts`: the hook now passes the chat service's Retry callback (`prepareFailedChatRunRetry`, registered at app startup next to the budget-resume callback). A task in a connected conversation whose endpoint restricts external execution then continues through the chat Retry path. Before, `requestTaskRetry` refused it (`retry_refused`) and the task stayed blocked. - `server/src/services/runner-lost-mid-turn-continuation.test.ts`: production-shape cases for a system assignment, an unlinked external chat sender, and a linked person's chat message. Negative cases: a deleted message, a message written by a run, a deleted linked message, and a system assignment whose Runner was not proven gone. - `doc/execution-semantics.md`: describes the rule. ## Verification - Red on `origin/master` with only the new tests: `cd server && npx vitest run src/services/runner-lost-mid-turn-continuation.test.ts` fails 4 of 16. The three new continuation cases return `{ requested: false }`. The deleted linked-message case returns `not_user_request` instead of `message_unavailable`. - Green with the fix: 16 of 16. Two more cases bind the task to a real restricted Slack endpoint and conversation. With the chat callback registered, the run continues and the Retry carries the chat action (18 of 18). Without the callback, the result is `retry_refused`, which was the behavior before for every connected conversation. - Related suites pass: `runner-lost-mid-turn-continuation.test.ts`, `heartbeat-native-remote-failure-retention.test.ts`, `workspace-recovery-retry-copyback.test.ts`, `governed-wait-runner-loss.test.ts` (116 tests). - `cd server && npx tsc --noEmit` passes. - Production evidence was read from staging run events through the read-only API: 3 of 17 losses at the 07:10Z swap took this path. After this change the next post-upgrade audit should show an automatic `task_retry` for every `native_continuation_requires_reconciliation` run whose refusal is `process_not_running`, including routine assignments and chat messages. ## Risks - Low. The change applies only after the Runner process is proven gone (`process_verification` / `process_not_running`), so no Runner can still run the turn. The turn is not replayed. The fresh session continues over the saved work, as in #21039. - The task can contain partial tool work from the lost turn. This is the same accepted risk as #21039. - Previous-release story: no persisted shape changes. The rule reads the wake's reason and comment id, which earlier releases already write. A run an older release already settled keeps its visible Retry. - This does not stop the Runner loss itself. That is a separate fix. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude (Anthropic) via Claude Code, with tool use (shell, file edits, test runs, read-only staging API reads). Model ID as attributed in the commit trailer: Claude Fable 5.2. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
4e519bd428 |
fix(chat): send writes with an empty responseText and point card declines to resolve-from-comment (#21050)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Chat agents call the Paperclip API through `call_api` and find
routes with `search_api`
> - #21035 added a pre-send refusal for writes that carry
`responseText`, and new summaries for the question cancel and withdraw
routes
> - Full chat cohort 5 (OpenCode with DeepSeek v4.1 flash) showed that
both changes went too far
> - The refusal blocked a valid write that had an empty `responseText`,
and the summaries hid the route that records a person's decline
> - This pull request narrows the refusal and corrects the guidance text
> - The benefit is that valid writes go through on the first try, and
agents settle a declined card instead of leaving it pending
## Linked Issues or Issue Description
Refs #21035, #21002
**What happened?**
- inbox-search.restore-archive: DeepSeek sent `responseText: {}` with a
valid body on `POST /api/issues/{id}/inbox-archive`. The refusal from
#21035 rejected 24 POSTs in 3 minutes. The model did not drop the key,
and the run was cancelled at the 240s window. In cohort 4 the same POSTs
went through on the first try.
- questions-and-approvals.reject-and-revise: the person said "Do not
publish it; just make a draft I can review." The CEO quoted the cancel
summary ("an agent withdraws its own pending card with POST
.../withdraw"), saw that the person created the publish card, and left
the card pending. The plan decision did not settle. In cohorts 3 and 4
the CEO declined the card with `resolve-from-comment` (resolver policy
`anyone`).
**Expected behavior**
- A write with an empty or default `responseText` is sent once. A write
whose `responseText` sets a window is still refused before it is sent.
- The guidance says that withdraw is only for the agent's own obsolete
card, and that a person's explicit decline or answer goes through
`resolve-from-comment` when the card's resolver policy allows.
**Steps to reproduce**
Run the two chat cases above with OpenCode and DeepSeek v4.1 flash on
master.
## What Changed
- `call_api` (`assertRunnerApiDispatchable`): the check reads the raw
`responseText` value. `{}` or `{ offsetBytes: 0 }` on a write is dropped
and the write is sent. A `responseText` with `limitBytes` or a later
offset is still refused. `saveResponse: false` on a write was already
accepted; a test now covers it.
- OpenAPI: the cancel and withdraw summaries say that withdraw is only
for the agent's own obsolete card, and that a decline or answer on the
person's explicit instruction uses `POST .../resolve-from-comment` when
the resolver policy allows. The `resolve-from-comment` summary says it
also records a decline of a card the person created.
- `AGENT_CHAT_DIRECTIVE`: a person's explicit decline in chat answers a
pending confirmation with reject, whoever created the card. Withdraw
never records the person's decision.
- `doc/runner-api-tools.md`: describes the narrowed refusal.
## Verification
- `server/src/services/native-runtime/runner-api.test.ts`: new test
sends `POST /api/issues/{id}/inbox-archive` with a valid body and
`responseText: {}`, `{ offsetBytes: 0 }`, `saveResponse: false`, or
both. The request is sent exactly once. Red before the fix (3 of 4 cases
refused), green after. The refusal test for a `responseText` with
`limitBytes` stays.
- `server/src/__tests__/openapi-routes.test.ts`: new test for the
cancel, withdraw and resolve-from-comment summaries. Red before, green
after.
- `server/src/__tests__/agent-conversations.test.ts`: new directive test
for the decline rule. Red before, green after.
- Also passed: `runner-api.integration.test.ts`,
`native-case-tools.integration.test.ts`, server `tsc --noEmit`. SDK
public input pins: `--check` reports no change.
- Paid run (OpenCode + DeepSeek v4.1 flash) on this branch for
inbox-search.restore-archive, questions-and-approvals.reject-and-revise,
gateways-profiles.rotate-token and budgets-costs.repair-cost-recording:
4 of 4 cells passed on
|
||
|
|
3c058c37e6 |
fix(retry): continue a connected task from saved work when its exact chat request cannot be repeated (#21038)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Tasks that come from a connected chat (for example GitHub pull request reviews) show a Retry button when a run fails > - The task projection offers Retry, but the Retry route sends a connected task to the chat retry owner, which only replays the exact failed chat request > - When that exact request cannot be replayed (the attempt already saved a result, its output was published, the latest run did not start from a chat message, or the requester's access changed), the route returns 409 and tells the person to check the connection by hand > - A Retry button that is shown must work, and recovery must not ask the person to inspect anything > - This pull request makes a person's Retry continue the task from its saved work when the exact chat replay is refused, the same way a new message on the task would > - The benefit is that the projection and the route agree, and blocked connected tasks become retryable after deploy with no migration ## Linked Issues or Issue Description No public issue exists. This is the bug description, in the fields of `.github/ISSUE_TEMPLATE/bug_report.yml`. **What happened?** On a hosted deploy, five blocked tasks that came from a connected GitHub conversation showed Retry (`interruption.canRetry: true`). Each click on Retry returned 409 `chat_failed_run_retry_not_authorized`. The message was "The exact failed chat request can no longer be retried safely. Check its current connection, access, and recovery state." or "This attempt already recorded a result. Send a new message on the task to continue from the saved work." The latest runs were legacy sandbox setup failures, native runs that needed reconciliation, a run that started from an issue monitor instead of a chat message, and a native run that already saved its result. Root cause: `admitTaskRetry` in `server/src/services/task-retry.ts` called the chat retry owner for every connected task and passed its exact-replay refusal straight to the person. The projection cannot predict that refusal. **Expected behavior** A shown Retry works. If the exact chat request cannot be repeated, Retry continues the task from its saved work, the same way a new message on the task would. Users do not get operator instructions. **Steps to reproduce** 1. Bind a task to a restricted chat conversation. 2. Make its latest run fail so that the chat retry owner refuses the exact replay. For example, use a run whose source is not a chat message, or a native run with an accepted result. 3. Click Retry (`POST /api/issues/:id/retry`). 4. The route returns 409 `chat_failed_run_retry_not_authorized`. **Paperclip version or commit** `master` at |
||
|
|
0dfbbdeb31 |
fix(chat): route company creation, decline retries and question cancellation correctly (#21035)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Chat agents follow the chat directive and discover API routes with search_api and call_api > - Full chat cohort 4 (OpenCode with DeepSeek v4.1 flash) showed four guidance gaps > - The missing-name rule overrode the board-only company-creation boundary; a repeated request after a decline did not post a new approval card; search_api showed the board-only question cancel route as open to agents (and withdraw as board-only); write calls carried `responseText` beside or instead of `body` > - This pull request fixes the guidance, the route authorization metadata and the pre-send refusal > - The benefit is that agents take the supported path the first time instead of asking for the wrong input, refusing a valid retry, or calling a route that returns 403 ## Linked Issues or Issue Description **What happened?** - onboarding.new-company: asked to "set up a new company for my website business", the CEO cited the missing-name rule (#21007) and asked for a business name with a question card. Creating a company is a board action, so the agent should send the person to the creation flow. - app-permissions.ask-before-action: the person declined a write card and then repeated the same request. The CEO asked the person to tell it to resubmit instead of posting a new approval card. The gateway already allows a fresh request after a new human message (`isNewHumanIntentAfterToolRejection`), but the decline text said "Do not retry the same call". - questions-and-approvals.multi-part-question: search_api listed `POST /api/issues/{id}/interactions/{interactionId}/cancel` as `board_or_agent`. The route rejects agents. The agent route, `/withdraw`, was listed as board-only although its route admits the card's creator agent and the task's assignee. - Several cases: write calls through call_api carried `responseText` with no body (for example "Received arguments: operationId, pathParams, responseText"). **Expected behavior** The agent links the company-creation flow; it posts a fresh approval card for a repeated request; search_api shows the true authorization of cancel and withdraw; a write with `responseText` is refused before it is sent, with the same guidance #21002 gave `saveResponse`. **Steps to reproduce** Run the four chat cases above with OpenCode and DeepSeek v4.1 flash on master. Refs #21002, #21007 ## What Changed - `AGENT_CHAT_DIRECTIVE`: the missing-name rule applies to work the agent creates itself and never comes before a board-only boundary. For a new company the agent links [Create organization](/onboarding) (no company prefix) and repeats the business details. A repeated request after a declined approval card is an explicit retry: make the same call so a fresh card is posted. A clarification question card puts candidate links in its description, because labels show as plain text. - Decline instructions in `tool-action-delivery.ts` and `tool-gateway.ts` say that a later request from the person for the same action is an explicit retry. - OpenAPI: `POST /api/issues/{id}/interactions/{interactionId}/cancel` is board-only and its summary names the `/withdraw` route for agents. `/withdraw` is no longer marked board-only, which matches its route. - `call_api`: a new write that carries `responseText` is refused before it is sent; the message says `responseText` never carries request fields and names the allowed body fields. Older saved calls still validate for recovery. `doc/runner-api-tools.md` is updated. ## Verification - `server/src/__tests__/agent-conversations.test.ts`: new directive tests for the board-only boundary order, the decline retry and card links. - `server/src/__tests__/openapi-routes.test.ts`: cancel is `board`, withdraw is `board_or_agent`, the cancel summary names withdraw. - `server/src/services/native-runtime/runner-api.test.ts`: a write with `responseText` is refused before sending (three window shapes), the same write without it is sent once, and saved calls still validate. - `tool-gateway.test.ts`, `tool-gateway-service.test.ts`, `issue-thread-interaction-routes.test.ts`, `tool-action-replay-intent.test.ts`: passed. `server` `tsc --noEmit`: passed. - Paid run (8 cells, OpenCode + DeepSeek v4.1 flash) together with the eval PR: see the PR comment for results. ## Risks - A model that used to send a write with `responseText` and a valid body now gets one refusal and must resend without it. Nothing is sent before the refusal, so there is no duplicate side effect. - The directive grows by a few sentences, which adds a small number of prompt tokens per chat turn. - Agents that read `/withdraw` as board-only before now see that they can withdraw their own card; the route already enforced who may. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use, shell execution and code editing. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
aa94939988 |
fix(native): continue a run whose remote Runner died mid-turn after a restart, without replaying the turn (#21039)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on a remote sandbox (Daytona) use a detached
`paperclip-runnerd`. After a server restart, recovery reattaches to that
Runner on the run's own sandbox lease
> - On staging, a deploy detached the controller and left the Runner
alive. The restart's reattachment then found the Runner process gone
with a provider turn in flight. The run failed with
`runner_remote_process_identity_unavailable` and the task waited for a
person
> - The run log said `startup reuse: environment (fresh) created`, so
the sandbox looked re-created. It was not. The reattachment resumed the
exact retained lease. The receipt reported the lease's original creation
> - `requestInterruptedNativeTurnRetry` refused every unsettled turn
(`provider_turn_unsettled`), and refused every system wake unless the
journal proved that no turn was issued. So this run got no automatic
continuation
> - This pull request continues such a run once in a fresh provider
session when the refused reattachment proved the Runner process gone. It
records that the turn's outcome is unknown and never replays the turn.
It also makes the environment receipt tell the truth
> - The benefit is that a restart that kills a Runner in the middle of a
turn no longer needs a Retry click
## Linked Issues or Issue Description
Refs #20853, #20923, #21023, #20278, #20627, #20828
**What happened?**
A staging deploy landed in the middle of a remote native Codex run on
Daytona. Shutdown detached the controller and logged "durable runner
left alive". The new server reattached through the run's original lease
(same lease row, same provider sandbox id, same directory). Its process
check then found the Runner gone: `process_verification` /
`process_not_running`, `runnerLifecycle: ready`, `runnerSettled: true`,
last diagnostic "WebSocket peer closed the connection", empty stderr.
The provider checkpoint still had an active turn. The run failed with
`runner_remote_process_identity_unavailable` /
`native_continuation_requires_reconciliation`, and the task went to
`blocked`.
**Root cause**
- `server/src/services/heartbeat/restart-interrupted-turn.ts`
(`requestInterruptedNativeTurnRetry`): an unsettled last turn always
returned `provider_turn_unsettled`, also when the refusal proved the
Runner process gone. A system wake (here `transient_failure_retry`)
continued only with `providerTurnIssued: false`.
- The sandbox was not re-created. The environment and lease listings
show one lease for the run, never replaced, and the final workspace
snapshot used the same provider sandbox. `environmentStartupReuse` read
`sandboxLeaseAcquisition: created` from the lease's first acquisition,
so the reattachment's receipt said `fresh / created`. The 46 s acquire
was the provider resume of that exact lease. The evidence available to
GET requests cannot say what stopped the Runner process (the resume
result is not kept). This pull request does not change that path.
- #21023 named this run as not covered. #20853 resumes a Runner only at
a settled, idle `ready` cut. Here the provider checkpoint has an active
turn, so #20853 stays fail-closed, as it should.
**Expected behavior**
The run continues once by itself in a fresh provider session over the
saved work. The turn whose outcome is unknown is recorded and is not
replayed.
## What Changed
- `restart-interrupted-turn.ts`: read the last
`native.remote_runner_recovery_failed` refusal once. When it proves the
Runner process gone, an unsettled last turn no longer blocks the
automatic Retry. Before it requests the Retry, the server records
`native.interrupted_turn_outcome_unknown` once (turn id, last turn
event, `replayed: false`, closed `runnerExit` fields). The Retry is the
existing one: saved-work recovery and stop proof first, then a fresh
provider session (`forceFreshSession: true`), once per chain.
- The same file: a system wake whose run is an automatic retry of owed
work (`retryOfRunId` set) also continues under the run's responsible
user once the refusal proved the Runner gone. Other system wakes (for
example a monitor) keep their policy.
- `packages/adapter-utils/src/startup-reuse.ts` and
`native-session-executor.ts`: a `reattach_remote_runner` recovery
reports its `environment` receipt as `reused / verified_reuse`. That
path only resumes the run's exact retained lease and never acquires one.
- Docs: `doc/execution-semantics.md` and `doc/run-log-events.md`.
Previous-release story: no persisted shape changes. The rule reads the
closed `remoteRunnerRecovery` fields that the previous release already
writes; the new test uses the staging refusal payload as written. A run
that a previous release already failed this way continues on the next
completion or restart hook call only; runs already settled stay as they
are and keep their Retry. The new event is additive.
## Verification
- Red on `origin/master` (
|
||
|
|
51d77648bc |
fix(native): keep a remote Runner alive when a failed turn start or event stream closes the run (#21023)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs on a remote sandbox (Daytona) start a detached `paperclip-runnerd`. After a recoverable failure, the server resumes the same run and reattaches to that Runner > - #20923 stopped a failed close from force-stopping an unreachable remote Runner. It asks for retention only when the session was not cancelled > - The runtime cancels the session itself when a turn start or the event stream fails ("Native session event consumption failed."). That cleanup sets the same flag as a user Stop > - So after a failed turn start, the close still sent `SIGKILL` to the Runner, and the same-run reattachment failed with `runner_remote_process_identity_unavailable`. The task waited for Retry > - This pull request separates failure cleanup from Stop. Failure cleanup still revokes the turn's output, but the close keeps the Runner for the same-run reattachment > - The benefit is that these runs continue without Retry. Stop and protocol integrity faults still stop the Runner ## Linked Issues or Issue Description Refs #20923, #20853 **What happened?** After the staging redeploy, five remote native runs failed with error `runner_remote_process_identity_unavailable` and error code `native_continuation_requires_reconciliation`. In four of them, the first attempt started after the deploy. Its `turn/start` failed with `runnerd did not report the provider turn identity`. The run log then showed `runner did not prove durable suspension before checkpoint` and `runner shutdown failed: Error: Durable recovery runner timed out.` The diagnostic `remote runner left alive unsettled for same-run reattachment` from #20923 did not appear. One second later the Runner exited (`runner_remote_process_exited lifecycle=ready`). The same-run reattachment 20–40 s later found `process_verification` / `process_not_running` with `runnerSettled: false`. **What did you expect to happen?** The failed close keeps the Runner alive, as #20923 intended. The same-run reattachment adopts it, and the run continues without Retry. **Steps to reproduce** 1. Start a remote native Codex run. 2. Make `turn/start` fail (for example the Runner does not report the provider turn identity in time), or let the event stream fail or time out. 3. The runtime cancels the session as cleanup. `HarnessNativeSession` treats that as a Stop, so `close` asks for `retainUnsettledRemoteRunner: false`, and the transport kills the Runner. **Root cause** `packages/paperclip-runner/src/native-session-runtime.ts` `attemptOptionalSessionCancellation` calls `session.cancel`. `HarnessNativeSession.cancel` (`packages/paperclip-runner/src/backends/harness-driver-backend.ts`) sets `#explicitlyCancelled`, and `close` used `retainUnsettledRemoteRunner: !this.#explicitlyCancelled`. The fifth run is different. A server shutdown detached its controller and left the Runner alive. Then the sandbox environment was acquired fresh (45.9 s) before the reattachment, and the Runner was gone in the middle of a turn. This pull request does not change that path. See Risks. ## What Changed - `NativeSession.cancel` accepts an optional `failureCleanup` field. - `attemptOptionalSessionCancellation` passes `failureCleanup: true`, except for a `NativeSessionProtocolIntegrityError`. - `HarnessNativeSession` tracks a Stop separately from output revocation. `cancel` with `failureCleanup` revokes the turn's output only. `cancel` without it, and `revokeTurnPublication`, record a Stop. `close` asks for retention when no Stop was recorded and no protocol integrity fault occurred. - Tests: three runtime cases with the real Codex backend (turn start fails, event stream times out, Stop then failure) and a new backend case. - Docs: `doc/execution-semantics.md` describes the rule. ## Verification - Red, on `master` with only the new tests: `cd packages/paperclip-runner && npx vitest run src/native-session-runtime.test.ts -t "remote Runner retention after a failed run"` fails 2 of 3 (`expected false to be true`): the close asked to kill the Runner after the turn start failed and after the event stream timed out. The Stop case passed. - Green with the fix: the same command passes 3 of 3. `src/backends/harness-driver-backend.test.ts -t "retain an unsettled"` passes 3 of 3, including the failure-cleanup case. That case also checks that the turn's result stays unpublished. - `npx vitest run src/native-session-runtime.test.ts src/backends src/drivers/codex src/live/runnerd-codex-transport-remote-retention.test.ts --maxWorkers=4`: 31 files, 814 tests pass. `codex-protocol-integrity.test.ts` confirms that an integrity fault still stops the Runner. - `npx tsc --noEmit -p packages/paperclip-runner/tsconfig.json` is clean. - `pnpm verify` exits 2 because `doc/execution-semantics.md` is a global CI input. CI supplies that coverage. - Not run: a live Daytona run. The change is in the controller library that runs on the server, so it needs no new Runner image. ## Risks - Low to medium. After a failed turn start or event stream, an unreachable remote Runner now stays alive instead of being killed. This is the state #20923 already accepted for other failed closes. The session stays quarantined, and the same-run reattachment or a Stop cleanup owns the Runner. - Stop is unchanged. A server Stop calls `cancel` without `failureCleanup`, so the close still stops the Runner, also when the stopped run later fails. A governed wait uses `revokeTurnPublication`, which also records a Stop. - A protocol integrity fault needs an operator and is not resumed on the same run, so it still stops the Runner. - Previous-release story: this change adds and changes no persisted state. A Runner that an older release already killed keeps its current named refusal and Retry. A deploy during a run changes nothing, because the decision is made in the closing process. - Not covered: a Runner that dies with its sandbox while the controller is detached for a restart, in the middle of a turn. Its journal is a settled `ready` cut with an active turn. #20853 resumes only idle `ready` cuts. This needs a separate fix. - No wait is added before `turn.start`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file edits, test runs, read-only staging API reads). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
cf1def0ca6 |
feat(inbox): rebuild compact cards and share task review controls (#20961)
## Thinking Path > - Paperclip helps people manage AI agents and their work. > - The inbox collects decisions, requests, and updates for the people in a company. > - People need a clear personal queue and a consistent way to answer each request. > - Repeated headings, inconsistent card actions, and separate review flows make that work harder. > - This pull request connects the inbox UI to the routing foundation in #20959. > - Compact cards use shared task controls and open supporting work in a side panel. > - People can answer, claim, transfer, archive, and inspect outcomes from one place. ## Linked Issues or Issue Description **What existing behavior does this improve?** The inbox cards, personal views, shared task controls, and decision history. **Subsystem affected** UI. This PR targets master and uses the routing foundation merged in #20959. **Current behavior** The inbox repeats headings and counts. Cards have different layouts and action placement. The requester can repeat the signed-in person's name. Review and question controls differ between the inbox and task thread. **Proposed behavior** Open Assigned to you by default. Show one count line above the cards. Use a compact card layout with consistent actions and supporting details. Show the requester's avatar and name only when they differ from the signed-in person. Share question controls, including a radio option for Other that reveals a text field when selected. **Reason and benefit** People can scan their queue and complete requests without losing the surrounding task or creating separate drafts for the same review. **Breaking changes** The default inbox view becomes personal. The all-assignee and shared views remain available. Card close menus distinguish personal archive and snooze from shared withdrawal. **Additional context** Refs #20959, #19363, and #19520. Refs #20268 for related actionable-request work. This stack adds personal routing and shared controls. ROADMAP.md includes Multiple Human Users and Agent Reviews and Approvals. ## What Changed - Add personal, unassigned, company, all-assignee, and archived inbox views. Add claim, transfer, personal close actions, receipts, and history controls. - Use compact decision cards with a single count line, one-pixel borders, and equal space above, below, and to the right of each primary action. - Open artifacts and synchronized confirmation/approval controls in a shared side panel. Share their revision drafts and action state with the task thread. Other requests keep one editable form on the inbox card while reviewing an artifact. - Make Other match the question options and reveal its field only when selected. Apply this to inbox and task question cards. - Hide self-requester labels, space agent avatars, and use the standard canceled status icon for withdrawn requests. - Use connector logos in connector cards. Keep the connector promotion background and sidebar border continuous. - Add company and project decision-owner controls and document the updated inbox behavior. ## Verification - The complete [GitHub Actions CI matrix](https://github.com/paperclipai/paperclip/actions/runs/43141559641) passes on `8d81a75e49b518fff6b6ff4c23dd67979b0b97ee` (Linux/Node 24: typecheck, build, general and serialized tests, Runner support, browser shards, and aggregate gates). Greptile is 5/5 on that exact head with zero open findings, and the security scan passes. - Local UI typechecking passes. Eight focused suites pass 382 tests: 236 inbox/review/question/connector tests plus all 146 composer tests. Regressions passed red to green for attachment links, prompt rendering, single-form artifact review, connector scope copy, accessible headings, and the composer test harness's missing theme provider. The final commit changes only that test harness; production code is unchanged from `59d015833f`. - Browser acceptance on the production implementation confirms preserved selected answers with one editable question form, accessible Markdown prompts, inert image references, accurate connector scope copy, native PDF downloads, image sidebar previews, and Open file requesting the attachment endpoint while preserving the inbox URL. Request reads and file responses were intercepted; no question answers were written. The preview is healthy on the final head. - Visual and interaction checks cover Other radio behavior in both themes and on mobile, the loaded Google Calendar logo, and 20px above, below, and to the right of the primary action across all 23 visible cards. The affected desktop/mobile navigation and sidebar specs pass locally and in the final CI browser matrix. - Real-editor browser checks confirm Ctrl/Cmd+Enter cannot bypass another person's confirmation. Cancel preserves the draft and unlocks silently; Continue submits once. Approval write failures remain visible. - `pnpm verify` returns exit 2 because DESIGN.md requires broader coverage; the final full CI matrix supplies every requested gate. Maps and Storybook catalog structural checks pass. The routing foundation #20959 merged after passing 249 database tests, server typechecking, full CI, Greptile 5/5, and security scanning. Its backend and SDK registry are preserved. - Demo data is retained for testing; database suites and browser specs use separate disposable databases. ## Risks - The inbox uses the routing contracts merged in #20959. Personal filters and history affect shared task entry points; the affected controls are included in local regression coverage and full CI. - Personal defaults and close-menu wording change inbox behavior. Existing resolver permissions still control shared actions. - Shared revision state and live invalidation affect two surfaces. Focused tests cover draft synchronization, action locks, and incoming updates. - Visual changes use the existing design tokens and component primitives. ## Model Used OpenAI GPT-6 through Codex. Used repository tools, code execution, and browser checks. This session does not expose a more specific runtime model ID or its context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal Paperclip ticket id - [x] I have run focused tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Scott Tong <scott@scottsmbpm5max.lan> Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
156453fe7c |
fix(recovery): continue runs that a server restart or replacement ended, without spending the failure budget (#21019)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A deploy replaces the server. The old server stops its running legacy runs at SIGTERM (`server_shutdown_interrupted`). The new server finds runs whose owner died without that drain (`process_lost`, "server may have restarted"). > - Each of these runs got its follow-up from the bounded transient retry. That retry spends the failure budget (2 attempts) and waits 30 to 60 seconds. > - So the first deploy spent one failure retry. A run that was already the last automatic retry got no follow-up at all. The task then rested on a manual Retry. > - A deploy is routine. An interruption by the server is not a failed attempt, and it must never leave a task waiting for a person. > - This pull request moves these runs to their own `server_restart` retry lane. The lane never spends the failure budget, starts the first continuation at once, and has its own bound. > - The benefit is that a task continues by itself through several deploys a day, and a real loop still stops and shows why. ## Linked Issues or Issue Description **What happened?** A zero-downtime deploy replaced a hosted server. In the 11 minutes around the swap, 11 legacy runs ended with `server_shutdown_interrupted` and 7 ended with `process_lost`. At least 7 tasks were left with no follow-up run and no scheduled retry, so each needed a manual Retry. Two of the stranded runs were already automatic retries. **Expected behavior** Each interrupted run continues once, automatically, on the surviving server, with its original wake. **Paperclip version or commit** Core commit |
||
|
|
8c761abae3 |
fix(environments): wait for a re-activating sandbox provider before refusing a lease resume (#21020)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - After a controller restart, a run whose remote Runner kept running
reattaches. First it resumes its exact sandbox lease through the sandbox
provider plugin (`resumeRunLease`).
> - #20982 made a plugin whose worker `initialize` timed out retry its
activation, so the plugin comes back seconds later instead of staying
`error`.
> - Chaos cell CR6 (#20983) shows the gap that remains. The reattaching
run asks for its lease during those seconds and gets
`PluginWorkerUnavailableError`. The setup retry covers that error only
for legacy runs, so the native restart-recovery run fails and needs
Retry.
> - This pull request makes the exact-lease resume wait, for a bounded
time, for the worker of a plugin that is `ready` (re-activating) or
recovering from activation timeouts.
> - The benefit is that a restarted run rides through a short plugin
re-activation instead of failing into Retry.
## Linked Issues or Issue Description
Refs #20982, #20983
**What happened?**
CR6 after #20982 (`w-20261112T041333-s9531`) passed in part. The plugin
re-activated 5 s after its `initialize` timeout, but both tasks finished
with `R3:retry_needed`. Run `a6b3f0f6` resumed its lease at 22:15:35,
inside that window. It failed with `setup_failed` /
`plugin_worker_unavailable` and then `agent_stop_unconfirmed`, because
`run-completion.ts` retries `PluginWorkerUnavailableError` only for
legacy runs (`runtimeMode === "legacy" && nativeSessionId === null &&
runnerInstanceId === null`).
**Expected behavior**
The resume waits for the plugin's worker while the plugin re-activates.
It resumes the same lease when the worker comes back. It still refuses,
as `PluginWorkerUnavailableError`, after the bound.
**Steps to reproduce**
1. Run chaos cell CR6: a K1 restart, then a stall of the replacement's
plugin worker during activation.
2. The reattaching run fails and needs Retry, although the plugin
recovers within seconds.
**Paperclip version or commit**
master at
|
||
|
|
a9a08d8169 |
fix(plugins): retry a plugin activation whose worker initialize timed out (#20982)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Plugins, sandbox providers such as Daytona among them, run as worker
processes that the server activates at boot. Activation sends the worker
an `initialize` RPC with a 15-second timeout.
> - When `initialize` timed out, the loader marked the plugin `error`
for the life of the server process. Only the next boot or an operator
retried it.
> - For a sandbox provider, that means every Daytona run on that server
fails. New runs fail with `lease_acquire_failed … plugin is currently
error`. Reattachments fail with "The exact sandbox lease cannot be
resumed by its verified provider". Retry hits the same dead plugin.
> - A worker that is slow or stalled for a moment at boot is transient,
not a broken plugin.
> - This pull request retries an activation whose `initialize` timed
out, with a bounded backoff. Meanwhile the server keeps serving, and
runs that need the plugin see a restarting worker, which setup retries.
Deterministic failures stay a permanent error.
> - The benefit is that a short plugin stall at boot no longer disables
a provider until the next deploy.
## Linked Issues or Issue Description
Refs #20966, #20857
**What happened?**
Chaos wave `w-20261111T232850-s9521` (cell CR4, first version) stopped
the replacement server's Daytona plugin worker while the server was
activating it. The worker's `initialize` RPC timed out after 15 s
(`INITIALIZE_TIMEOUT_MS`). `plugin-loader.ts` then called
`lifecycleManager.markError`, and the plugin stayed `error`. Task t03's
reattachment failed in `resumeRunLease`, and its Retry did not start a
new run. Task t04 failed 18 s after launch with `lease_acquire_failed`
("…plugin is currently error"), before its own fault.
**Expected behavior**
An activation timeout is retried, and the plugin ends `ready` when the
worker answers. Runs that need the plugin meanwhile are retried within
their bounded setup budget. A bad manifest, a worker that exits during
`initialize`, or a version mismatch is still a permanent error.
**Steps to reproduce**
1. Start the server.
2. Stop (SIGSTOP) a plugin worker for more than 15 s while the server
activates it.
3. Resume the worker. The plugin stays `error` until the next boot.
**Paperclip version or commit**
master at
|
||
|
|
50b79d711a |
fix(runner): keep call_api writes carrying their body (#21002)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native agents use the `call_api` tool when no dedicated tool covers an API operation > - #20606 added an optional `saveResponse` flag to the shared `call_api` schema for GET reads > - After that change, OpenCode agents with DeepSeek v4.1 flash often sent write calls without `body` > - Body-less POST calls reached the routes and failed with 400; agents then repeated the same call many times > - This pull request moves the save option to its own read-only tool and refuses body-less writes before they are sent > - The benefit is that the write schema returns to its earlier shape, and a missed `body` gets one clear, actionable error ## Linked Issues or Issue Description Refs #20606 **What happened?** In a full chat-eval cohort on master (OpenCode, DeepSeek v4.1 flash), write calls through `call_api` arrived without `body`. The model's own reasoning planned a body, but the emitted calls had only `operationId`, `pathParams`, `query` and `saveResponse`. About 100 refused PATCH calls listed exactly those arguments (the cohort before #20606 had none with `saveResponse`). 72 POST calls reached the server with no body and got 400 (0 in the earlier cohort). One case sent the same body-less resolve call 21 times. **Expected behavior** Write calls carry their request fields in `body`. When a body is missing, the agent gets one clear error before anything is sent, and it can correct the call. **Steps to reproduce** Run the OpenCode chat cases that resolve approvals, edit queued messages, delegate with context, edit task properties or start preview services on master after #20606. Count `call_api` writes without `body` and POST 400 responses. **Additional context** The tool schema that OpenCode sends to the model is intact: `body` is present. A short-context replay with the same schema did not reproduce the omission (direct provider calls and local OpenCode runs: every write carried `body`, with or without `saveResponse`). The omission appears in long sessions, and it correlates with the added read-only flag. This change removes that flag from the write schema and adds a safety net for the remaining cases. ## What Changed - Remove `saveResponse` from the advertised `call_api` schema and description. The description now says that request fields go only in `body`, and the `body` description says when it is required. - Add a separate read-only `save_api_response` tool (`operationId`, `pathParams`, `query`). It accepts only GET operation IDs, has no body field, and uses the existing `saveResponse` export path with the same access, sensitivity, budget and integrity rules. - Refuse a write before dispatch when its body is missing: PATCH with a required body (as before), and POST or PUT whose JSON request schema names required fields. The error names the operation, the allowed fields, the received arguments, the shape to resend, and that `query`, `responseText` and `saveResponse` never carry request fields. - Keep server-side acceptance of `saveResponse` on `call_api`, so saved calls and receipts from earlier runs stay valid for recovery. - Regenerate the runner contract, catalog, seeded fixture and manifest. Update `doc/runner-api-tools.md`. ## Verification - Red at master sources: the 3 replayed recorded writes (POST resolve-from-comment, PATCH issue, POST comment, each with `query: {}` and `saveResponse: false` and no body) were sent to the route instead of being refused, and the schema test found no `save_api_response` tool. 4 tests failed. - Green with this change: `server/src/services/native-runtime/runner-api.test.ts` 2105/2105 passed; `runner-api.integration.test.ts` 61/61 passed; `runner-bridge-tools.test.ts`, `task-document-key-contract.test.ts` and `runner-api-artifacts.test.ts` passed; runner `protocol-action-contracts.test.ts` 70/70 passed; server `tsc --noEmit` passed. - `runner-api-artifacts.integration.test.ts` now saves through `save_api_response` and checks that a write operation ID or a body is refused. Its download step needs Linux; it fails on macOS at master too, so CI covers it. - Paid E2E, OpenCode + DeepSeek v4.1 flash, [run 43138806990](https://github.com/paperclipai/paperclip/actions/runs/43138806990) on this fix: 5/5 affected cases passed (questions-and-approvals.reject-and-revise, steering.edit-queued-message, task-coordination.delegate-with-context, tasks.edit-properties, workspaces.preview-service). Body-less write refusals went from 30 to 0, and body-less POST 400s went from 48 to 0, compared with the earlier cohort on the same cases. The approval resolve call was sent once (it was sent 24 times before). Every logged write carried a body; the remaining non-2xx writes are the cases' intended stale-edit and failing-fixture responses. - Run-history redaction also covers `save_api_response` arguments; a regression test fails on the earlier source and passes now. ## Risks - Agents that learned `call_api` + `saveResponse: true` in a running session will see a schema refusal and must use `save_api_response`. Saved calls keep working for recovery. - The new POST/PUT refusal applies only when the JSON request schema names required fields, so writes whose body is optional are still sent. - One more small read tool is advertised when API tools are enabled. ## Model Used Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use, shell execution and code editing. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
faa78dd818 |
fix(agents): keep a managed instructions bundle managed when a restarted volume-free server saves another setting (#20997)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Agent settings are saved through `PATCH /api/agents/:id`. On a
volume-free server, a save is refused when the agent's instructions are
external, because files outside the managed root do not survive a
restart
> - A managed bundle saves its instruction paths as absolute paths under
the home of the server that wrote them. A replacement volume-free server
can start with a new home
> - After such a restart, any `adapterConfig` change re-derived the
bundle mode from the saved `instructionsFilePath`. That path is outside
the new managed root, so the bundle was read as `external` and the save
failed with 422 "Volume-free servers require managed agent instructions"
> - This pull request keeps a managed bundle managed when the save keeps
its saved `instructionsFilePath`, and rewrites its paths under the
current managed root
> - The benefit is that a board user can change one setting (for example
`idleTimeoutMs`) after a restart, and a real switch to external
instructions is still refused
## Linked Issues or Issue Description
**What happened?**
In restart-chaos wave `w-20261112T012841-s7000` (volume-free mode,
Daytona, Codex, restarts K1 and K15 with a fresh server home), the board
sent `PATCH /api/agents/:id` with
`{"adapterConfig":{"idleTimeoutMs":301001}}`. All 3 tasks of the W2C
workload got 422 "Volume-free servers require managed agent
instructions".
**Expected behavior**
A partial `adapterConfig` change of an agent with managed instructions
succeeds on a restarted volume-free server.
**Steps to reproduce**
1. Create an agent with a managed instructions bundle on a volume-free
server.
2. Restart the server with a new `PAPERCLIP_HOME`.
3. Send `PATCH /api/agents/:id` with
`{"adapterConfig":{"idleTimeoutMs":301001}}`.
4. The server returns 422.
**Why Codex looked different from Claude**
It is not the adapter. The Claude wave that passed
(`w-20261112T010036-s6801`) had no faults, so its PATCH ran on the
server that wrote the bundle. The Codex wave sent its PATCH after K1 and
K15 restarts with a new home. The new test fails the same way for a
Claude agent after a fresh-home restart, and passes for both adapters on
the original server.
**The PATCH contract**
`adapterConfig` merges into the saved config unless the request sets
`replaceAdapterConfig: true` (validator in
`packages/shared/src/validators/agent.ts`, route in
`server/src/routes/agents.ts`). The UI full-config save sends
`replaceAdapterConfig: true` with the whole config. The merge was
already correct. The fault was in
`syncInstructionsBundleConfigFromFilePath`, which the route applies to
the effective (merged) config: it classified the bundle only from the
absolute file path and the current managed root. The contract does not
change.
## What Changed
- `server/src/services/agent-instructions.ts`
(`syncInstructionsBundleConfigFromFilePath`): when the saved bundle is
managed and the effective config keeps the saved `instructionsFilePath`,
keep it managed with the root that `deriveBundleState` already resolves
(the current server's managed root) and the saved entry file. A
different `instructionsFilePath` is classified as before.
- `server/src/__tests__/agent-volume-free-config-patch.test.ts`: route
tests with embedded Postgres for a Codex agent and a Claude agent: a
one-setting change after a fresh-home restart, the same change on the
original server, and a real switch to an external path, which is still
refused.
- `doc/DEVELOPING.md`: describe how a volume-free server keeps a managed
bundle across a new home.
## Verification
- Red on `origin/master` (`9dfdeaad86`), `cd server && npx vitest run
src/__tests__/agent-volume-free-config-patch.test.ts`:
- `× changes one codex setting after a fresh-home restart and keeps
managed instructions`
- `× changes one claude setting after a fresh-home restart and keeps
managed instructions`
- `AssertionError: {"error":"Volume-free servers require managed agent
instructions"}: expected 422 to be 200`
- The 4 other cases passed (the original server, and the refused
external switch).
- Green with the fix: the same file passes 6/6, together with
`agent-instructions-service`, `agent-instructions-routes`,
`ephemeral-server-contract` and `agent-configuration-save` (65 tests).
- The restart-chaos W2C workload (`tests/restart-chaos/rewake.ts`)
already sends this PATCH after K1/K15 restarts and caught the fault as
`T:config_changed_before_wake`. No new chaos cell is necessary.
## Risks
- Low risk. The change applies only when the saved bundle is managed and
the save keeps the saved `instructionsFilePath`. Then the paths are
rewritten under the current managed root, which is the root that bundle
reads already use.
- The volume-free guard is unchanged. A save that sets an
`instructionsFilePath` outside the managed root is still refused (tested
for both adapters).
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, test runs).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
45170b0cac | Reuse Runner plugin capability checks (#20987) | ||
|
|
5cd5adfccc |
fix(native): link a person-owned agent blocker to the agent's open question whatever owner kind the model reports (#20969)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - #20931 lets a person's answer continue a task that its agent blocked
while it waited for that answer
> - That link is recorded only when the blocker's unblock descriptor is
bound to the board
> - Codex reports such a wait as `paperclip_block` with `owner: {"kind":
"user", "name": "Board"}` and scope `task_wide`. The evidence classifier
treats only `kind: "board"` (not a PRP owner kind) as board-owned, so
the arbiter binds this person-owned blocker to the reporting agent, and
no link is recorded
> - So every Codex W2Q task still loses the answer
(`issue_not_in_progress`) and needs Retry
> - This pull request classifies a `user` owner as a person
(board-bound) and records the link from the reported owner kind, not
from the spelling of the owner
> - The benefit is that the person's answer reaches the agent whatever
owner shape the model uses
## Linked Issues or Issue Description
Refs #20931
**What happened?**
Restart-chaos wave w-20261111T230717-s5201 on master
|
||
|
|
38501b1170 |
Add turn-scoped attachment downloads to native Runner (#20956)
## Thinking Path > - Paperclip manages AI agents and their work. > - Native agents open input images with a file reader. > - Plugin Runner tasks use an authenticated Runner connection. > - Runner needs a way to fetch an authorized file from the tenant app. > - The tenant grants access to the exact attachment for the active run. > - Runner downloads and verifies it before it returns a local path. ## Linked Issues or Issue Description Refs #19740. Builds on the attachment authorization and command transport added in #20939. ## What Changed - Add the PRP v3 `runner.attachment.download` command for native workspace providers. - Issue short-lived grants for exact authorized wake and historical attachments. Recheck the active run, source, size, and SHA-256 at the tenant endpoint. - Download current-wake inputs before provider dispatch. Download historical files when the authorized chat reader asks for them. Preserve SSH and sandbox command staging as a fallback. - Run HTTPS transfers off the Runner control loop. Report completion with one PRP event per transfer. Drop pending results on cancellation and clear files at turn settlement, authority rotation, and restart. - Grant restricted Codex sessions read-only access to the attachment directory. ## Verification - 69 targeted server attachment tests passed. They cover wake and historical downloads, authorization failures, and command fallback. - 22 focused historical attachment tests passed, including grant redemption under a held database lock. - Six focused Rust attachment tests passed, including a source-pinned Codex sandbox check. The sandbox could read the staged file but could not modify it or read adjacent Runner state. - The real Runner transport test confirmed one durable command and one completion event for a transfer. Restart-continuation tests confirmed fresh attachment paths appear before dispatch. - Server and Runner TypeScript checks, Rust formatting, and PRP generated schema checks passed. - Required GitHub CI passed on `7a33752bd7`. [CI run](https://github.com/paperclipai/paperclip/actions/runs/43122967837) - Greptile scored the same revision 5/5 with no unresolved review threads. [Review](https://github.com/paperclipai/paperclip/pull/20956#issuecomment-6808582134) ## Risks - Saved provider and Runner identities keep their existing formats. A previous Runner falls back to command staging where the target supports it. A restart continuation refreshes temporary attachment paths before dispatch. - Plugin Runner tasks require a compatible Runner ROM and outbound HTTPS access to the configured tenant origin. - Live agent-computer qualification remains a release check. ## Model Used OpenAI GPT-6 through Codex, with repository inspection, shell tools, and local test execution. Exact variant and context window are unavailable in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #123` / `Closes #123` / `Refs #123` or (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
2539004201 |
fix(ephemeral): start a restarted volume-free server over the Runner controller files it wrote itself (#20952)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Cloud servers run volume-free (`PAPERCLIP_EPHEMERAL_SERVER=true`). Postgres and object storage hold the durable state. A startup gate refuses legacy volume data that was not migrated. > - The gate reads one family of data from disk: Runner controller files under `runtime/paperclip-runner/durable-sessions/`. It treats a controller file that the database does not prove as legacy data. > - A volume-free server also writes these files during normal work. A warm follow-up restores the previous run's checkpoint as a full `durable.control-plane-state.v2` file. The file stays full until the new controller registers its database checkpoint writer. > - When the server crashes in that window and restarts on the same filesystem, the gate finds its own file and refuses to start. Every later start refuses the same way. This is a full outage that no Retry can fix. > - This pull request gives the gate a durable marker. A volume-free server claims an empty state directory before any controller can write. Files in a claimed directory belong to a volume-free server and do not block startup. > - The benefit is that a crash restart of a volume-free server starts again. Genuine legacy data keeps its protection, and the refusal now names the run, task and file. ## Linked Issues or Issue Description **What happened?** The first restart-chaos waves in volume-free server mode (harness flag `--ephemeral-server`, #20930) found this outage. Wave `w-20261111T212429-s6402` (Claude through ACPX, Daytona warm sandboxes) did a fresh-home deploy and then a `kill -9` of that server. The replacement server exited with `Volume-free server data is not eligible: native_controllers_require_migration`. A second start at +608 s exited the same way. The server stayed down. **Root cause** `hasLocalNativeControllerObligations` in `server/src/services/native-runtime/native-controller-migration.ts` scans every Runner session in the state directory. The kept instance shows the cause. The session's newest run `4a4d7ca1` was `running` (admitted 3 s before the kill). Its controller root held the full `paperclip.runner.durable.control-plane-state.v2` file of its warm predecessor `9c13b7f8`. A controller writes that full file when it opens and replaces it with the `external-control-plane-state.v1` marker only in `setCheckpointPersistence`. `preservedActiveCheckpoint` accepts only the marker, so the gate returned `true`. A Codex wave with the same fault sequence passed because no warm follow-up was in this window at the kill. Claude W2 cells start a new warm follow-up every 15 to 35 s, so the window is open much more often. **Can staging hit it?** Yes, with a restart that keeps the container. Railway documents that a manual Restart "restarts the process within the deployment's container", so the filesystem stays. Railway does not document whether an automatic crash restart reuses the container. A redeploy makes a new container, so it does not hit this. **Expected behavior** A volume-free server starts again after any restart. State that it wrote itself is disposable, because the database checkpoint is the durable record. Data that a volume-backed server wrote keeps its current protection. ## What Changed - Add `server/src/services/native-runtime/volume-free-controller-state.ts`. `claimVolumeFreeControllerState(true)` writes `.volume-free-owner.json` atomically (temporary file, fsync, rename, directory fsync) when the state directory is absent or empty. It returns `owned` for a directory that has the marker and `unclaimed` for a directory with other files. `claimVolumeFreeControllerState(false)` removes the marker. - `server/src/index.ts` calls the claim right after the volume-free configuration check. This is before any controller or recovery can write. A server started without volume-free mode releases the claim, because it may write files that exist only on its disk. An `unclaimed` result logs a warning. - `hasLocalNativeControllerObligations` ignores the marker entries. It returns no obligation for a claimed directory. It now delegates to `findLocalNativeControllerObligation`, which returns the run, task and file (or `database_checkpoint`). - `assertEphemeralServerData` adds that detail to the `native_controllers_require_migration` refusal. The error says which run and file the database does not prove and why startup refuses. - Document the marker in `doc/DEVELOPING.md` ("Runner controller state on a restarted volume-free server"). ## Verification - Red, before the gate reads the marker: `pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/native-controller-migration.integration.test.ts -t volume_free_owner` failed with `AssertionError: expected true to be false` at the first gate check. The fixture is a running warm run whose root holds a full `durable.control-plane-state.v2` file, the same shape as the incident. - Green: the same test passes. It checks: claimed directory gives no obligation; release gives the obligation back; a non-empty unclaimed directory stays blocked; the refusal names the run; the controller file is never changed or removed. - `ephemeral-server-data.test.ts`, `ephemeral-server-contract.test.ts` and all 57 cells of `native-controller-migration.integration.test.ts` pass. Server `tsc --noEmit` passes. - Probe on a copy of the kept instance database and the killed server's state directory: the unmodified gate returned an obligation for run `4a4d7ca1` (source file `.../7dbf2d5c.../control-plane/control-plane-state.json`). - Chaos: the s6402 shape (`--ephemeral-server --adapters claude --workloads W2 --faults K1,K2,K15 --k15-fresh-home --seed 6402`) runs on this branch merged with #20930. The result is added in a comment. ## Risks - A filesystem that a volume-free server claimed and that a pre-marker, volume-backed server then wrote is not detected. This needs a downgrade and a mode change on one disk. A current server in volume mode removes the marker, which closes the normal path. - A volume-free server on a directory written before this change (no marker) keeps today's refusal. A new deploy makes a new container, so staging claims the empty directory on first start after this deploy. - Other scanners of the state directory filter on the 64-hex session pattern, so the marker file does not change them. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, file edit, test runs). Default context window, extended reasoning on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
d1e4109df1 |
fix(runner): keep a connected adopted Runner through a failed liveness probe (#20965)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - After a controller restart, the controller adopts the remote Runner
that kept running in the sandbox. A monitor then checks the Runner's
liveness every 250 ms through the environment plugin.
> - #20940 and #20945 made a stalled plugin probe "unknown" for the
launched Runner's monitor, reattachment and launch. The adopted Runner's
monitor still failed the run on the first probe that failed without an
answer.
> - A short plugin stall after adoption therefore failed a healthy run
with `native_adopted_runner_identity_unverifiable`.
> - This pull request uses the evidence that is available: while the
adopted Runner holds its authenticated connection to the controller, a
liveness check that fails without an answer does not fail the run, for
at most the reconnect grace.
> - The benefit is that a healthy adopted Runner survives a short plugin
stall. A check that reports the process gone still fails the run at
once.
## Linked Issues or Issue Description
Refs #20940, #20945
**What happened?**
`#watchAdoptedRunner` in
`packages/paperclip-runner/src/live/runnerd-codex-transport.ts` called
`failTransport("native_adopted_runner_identity_unverifiable: runner
liveness could not be revalidated")` on the first `isAlive()` rejection.
For a remote adopted Runner, `isAlive()` is a plugin
`environmentExecute` RPC. A plugin stall makes it reject, even while the
Runner holds its authenticated connection and works normally.
**Expected behavior**
A rejection is not an answer. While the Runner holds its authenticated
connection, the monitor keeps checking. The bound is the reconnect
grace. A check that returns `false` (the process is gone) still fails
the run at once.
**Steps to reproduce**
1. Adopt a remote Runner after a controller restart.
2. Stall the plugin worker, so that the liveness probe rejects.
3. Within 250 ms the run fails with
`native_adopted_runner_identity_unverifiable`.
**Paperclip version or commit**
master at
|
||
|
|
827bd21017 |
fix(recovery): retry lost-lease preparations instead of holding them for reconciliation (#20936)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Run recovery restarts work that stops for infrastructure reasons,
and keeps saved files safe
> - A legacy Runner run can lose its controller lease while it prepares
the workspace, before any provider work starts (`controller_lease_lost`
with `executionRecovery: { kind: "bootstrap", providerWorkStarted: false
}`)
> - Such a run has no unsaved provider state, but recovery still blocked
the task: either a saved-files hold that ends in `source_unavailable`,
or an immediate "No live execution path" block after one failed
continuation retry
> - Recovery must work without a person, and a shown Retry must work
> - This pull request lets lost-lease preparations use the existing
interrupted-preparation settlement, and lets the bounded stranded-task
sweep retry a continuation that failed before provider work
> - The benefit is that these tasks retry automatically, and tasks that
an older release already blocked recover after deploy
## Linked Issues or Issue Description
Refs #20924 (fixes the lease renewal starvation that caused the lost
leases; this PR fixes what recovery did afterwards).
**What happened?**
A staging burst of lost controller leases stopped many legacy Runner
runs while they prepared the workspace. No provider work started. Most
of the affected tasks then became blocked. One shape kept a saved-files
hold (`legacy_execution_requires_reconciliation`). Its file recovery
ended `exhausted` with `source_unavailable`, because the workspace had
already retired the prepared epoch back to the last save. The hold never
settled. The other shape was a continuation retry that also lost its
lease. The release path blocked the task at once with "No live execution
path", although `controller_lease_lost` is a transient cause with a
3-attempt budget in the stranded-task sweep.
**Expected behavior**
A run that fails before provider work starts retries automatically,
within a bounded budget. It never needs reconciliation. If the prepared
workspace is gone, the task continues from the last committed save.
**Steps to reproduce**
1. Start a legacy Runner run with a remote workspace.
2. Make the controller lose its lease while the run prepares (before
provider dispatch).
3. Let restore recovery record a saved-files hold, and let the workspace
retire the prepared epoch.
4. See that the hold stays `resolved/blocked` with `source_unavailable`,
and the task stays blocked.
**Paperclip version or commit**
master at
|
||
|
|
c45f54ee74 |
fix(native): stop or release unclosable idle warm Runners on environment delete, and say why a close failed (#20957)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs keep a warm Runner in a Daytona sandbox between turns of
a task, so the next turn starts fast
> - Each idle warm Runner has an in-memory owner in the controller.
Deleting an environment closes these owners before it destroys the
sandboxes
> - In every warm chaos wave today, some owners never closed. The server
logged only `{"failed":2}`. Environment delete returned 409 forever, and
all sandboxes of the wave stayed `started` until their 60-minute TTL
> - On staging, that is leaked sandboxes (cost) and owners that block
the work after them
> - This pull request logs and reports the exact close error, stops an
unsettled idle Runner when its environment is deleted, and releases an
owner whose sandbox is already destroyed
> - The benefit is that environment delete finishes or names the exact
run and error, and every warm chaos wave now grades a clean shutdown
## Linked Issues or Issue Description
**What happened?**
After warm Daytona chaos waves (W2, W2Q, W2C cells, `--faults K0`) on
Codex, Claude and OpenCode, the server logged `idle native session
cleanup failed before controller shutdown {"idleSessions":{"failed":2}}`
with no reason. `DELETE
/api/environments/:id?destroyReusableSandboxLeases=true` returned 409
"warm native runner did not shut down cleanly" on every retry. Eight
sandboxes per wave stayed `started`.
**Root cause.**
`closeIdleWarmNativeSessions`
(`server/src/services/native-runtime/native-session-executor.ts`)
dropped the close error (`catch {}`) and counted it. The owners that
failed are the W2Q owners. A W2Q task parks a question and is answered
by a later run on a fresh sandbox. With the reason logged, a wave shows
two causes for the same owner:
1. The owner's close cannot settle (`native_session_close_unrecoverable:
runner did not durably suspend before checkpoint`, the governed-wait
shape from #20844). Only a successor reset could force a stop
(`discardUnsettledSession`). An idle sweep or environment delete had no
forced stop.
2. The owner's sandbox lease was already destroyed by the
pending-cleanup sweep after the successor moved to a fresh sandbox
(#20863). Every close then fails with `Daytona sandbox lease <id> is no
longer active.`, because the stop RPC has no sandbox to reach. The owner
stays registered for ever.
The Runner close memoizes its rejection, so every later attempt fails at
once with the same error. This is not the "controller sequence must be
contiguous" exit (#20946): no close failure in these waves names a
sequence mismatch.
**Expected behavior**
An idle warm owner with no turn in flight never blocks environment
delete for ever. Every failed close names its run and its error.
**Steps to reproduce**
1. Run `tests/restart-chaos/main.ts --env daytona --daytona-warm
--adapters claude --workloads W2,W2Q,W2C --faults K0 --concurrency 8
--tasks 8`.
2. Read `<wave>/sandbox-sweep.json`: `environmentDeleteBlocked` is set
and the sandboxes are `started`.
## What Changed
- `closeIdleWarmNativeSessions` now returns `failures: [{ runId,
issueId, environmentId, providerLeaseId, reason }]`. The reason is the
close error with its code, its settlement record and its cause. It also
logs `idle warm native sessions did not close` with that list. The
idle-timeout close logs its error too.
- `closeWarmNativeSessionsForEnvironment` takes `forceStopUnsettled` and
`db`. The environment delete route sets both. When the graceful close
fails, Core retires the owner as #20844 does: one confirmed Launcher
`stop`, the credential snapshot, `complete`, then instruction
collection. A failed forced stop is added to the reason.
- When retirement still fails and the owner's sandbox lease has a
`destroyed` termination receipt, Core releases the owner. It does not
contact the sandbox. It logs `warm native session's sandbox was already
destroyed; released its owner` with the close error.
- The environment delete 409 now names each run and its reasons, and
`details.nativeRunnerSessions.failures` lists them.
- Idle timeout, controller restart and tenant sleep do not force a stop.
Their conversation can still be resumed, so they only log the reason.
- Chaos harness: new check `warm-shutdown.ts`, "no leaked sandboxes /
clean warm shutdown". Every warm wave (`--daytona-warm` or `--lifecycle
warm`) runs it. It fails the wave on a blocked delete, on a sandbox left
running after cleanup, or on a warm session that did not close at
environment delete or controller shutdown. It writes
`<wave>/warm-shutdown.json` and a section in `SUMMARY.md`.
- Docs: `doc/plugins/PLUGIN_RUNNER_EXECUTION.md` and the chaos README.
## Verification
- RED on master: `native-session-executor.test.ts -t "stops an idle warm
owner whose close cannot settle"` fails: `expected { closed: 0, busy: 0,
failed: 1 } to match object { closed: 0, failed: 1, failures: [...] }`
(a bare count, no reason). `environment-routes.test.ts -t "names the
warm runner session"` fails: the 409 does not name the run. On the first
fix commit, `-t "sandbox lease is gone"` fails: the lease state is never
read, and the owner stays.
- GREEN: those tests pass. The executor suite, environment routes, run
control, recovery, heartbeat process recovery, instance settings, the
idle-sleep, eviction, release-resume, restart-recovery and
warm-checkpoint integration suites pass (11 files, 2471 tests).
`tests/restart-chaos` unit test `warm-shutdown` passes. Server and
harness `tsc --noEmit` pass.
- Daytona warm waves (Claude, W2/W2Q/W2C, K0, 8 tasks). The runner image
`sha256:e04d9ce4…` matches
|
||
|
|
9c56e5465c |
fix(native): keep the agent's own unanswered question open after a turn ends without a report (#20934)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - An agent can ask the person a question and wait for the answer. Native task status comes from the server's status arbiter, not from model prose > - When a native turn ends without `paperclip_finish`, the Runner sends a default report: `done`, objective satisfied, with the agent's final message as the only evidence > - A later comment from the person moves an open question into history (#15392), so the question no longer gates completion > - Together these close a task as `done` while the agent is still waiting for its own answer and has produced nothing. No Retry appears and nothing flags it > - This pull request keeps such a task `in_review` on the agent's own unanswered question when the only completion evidence is the agent's final message > - The benefit is that a "waiting for your answer" turn can no longer become a silent false completion ## Linked Issues or Issue Description Refs #15392 **What happened?** Restart-chaos wave on master |
||
|
|
6cfb1f3d87 |
fix(environments): resume a warm sandbox that its finished predecessor is still releasing (#20953)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A warm Paperclip Runner session keeps its Daytona sandbox between runs. The next run of the same task resumes that sandbox through a `reuse_by_environment` lease. > - Acquisition only resumes a lease that is `released`, `retained`, or held by the same run. A finished run releases its lease only after it saves its instruction and workspace files, which takes several seconds. > - A board wake that arrives in that window does not see the warm lease. It creates a fresh sandbox. The session's durable backup cannot be restored there, because its source lease is still held. The run fails with `native_session_recovery_failed: runner_harness_state_mismatch: backup_without_reusable_lease` and the task needs Retry, which starts a new provider session. > - This pull request makes acquisition wait, for a short bounded time, for a lease that a recently finished predecessor is still releasing, and then resume it. > - The benefit is that the follow-up continues the same warm session in the same sandbox, with no Retry, no restore and no lost provider context. ## Linked Issues or Issue Description **What happened?** Restart-chaos wave `w-20261111T205259-s6301` (Codex, Daytona warm, volume-free server, `--ephemeral-server` from #20930), task t01. The board re-wake run `e7129ec7` started at 20:57:44. Its predecessor `5cf8c306` reported `run.terminal` at 20:57:38 but saved its instruction copy until 20:57:51, so its warm lease was still `active`. At 20:57:53 the new run recorded `startup reuse: environment (fresh)`. It then failed with `runner_harness_state_mismatch: backup_without_reusable_lease` (error code `native_session_interrupted`). Recording the `workspace_finalize` operation then failed with a Postgres deadlock, and the run settled `native_finalization_missing`. Retry run `79aa18ec` worked with a fresh provider session. The K1 restart in that wave was at 20:55:32 and is not the cause: six warm runs after it reused the sandbox normally. **Root cause** `reusableLeaseCanBeResumed` in `server/src/services/environment-runtime.ts` excludes an `active` lease held by another run. That rule is correct for a running predecessor. For a predecessor that has already ended, the `active` state is a short release tail. The new run creates a sandbox, and `retiredSourceAuthorizesNativeBackupRestore` (`native-retired-backup-source.ts`) correctly refuses to restore the backup from a source lease that is not yet `released` or `retained` (`idleWarmSourceRetiresBackup`). The executor's last branch (`native-session-executor.ts`, host preparation, created allocation) then throws `backup_without_reusable_lease`. **How often** Seen in 2 of 361 retained wave result directories on this machine (`s6301` today, `s5201` earlier). It needs a board wake within a few seconds of the predecessor's terminal event, so it is timing dependent. ## What Changed - `server/src/services/environment-runtime.ts`: add `waitForSettlingReusableLease`. Before the plugin and built-in sandbox paths select a reusable lease, acquisition polls (250 ms) for up to 20 s while a matching `reuse_by_environment` lease is `active` under a different run that is `succeeded`, `failed`, `cancelled` or `timed_out` and finished in the last two minutes. It then continues with the normal selection, which resumes the released lease. - A running predecessor causes no wait (unchanged: the new run gets a new sandbox). A predecessor that ended more than two minutes ago and never released causes no wait. - `server/src/__tests__/environment-runtime.test.ts`: two red/green cases (succeeded and failed predecessor release the lease 400 ms after the successor starts acquiring) and one case for an old unreleased lease (no wait, new sandbox, old lease untouched). - `doc/execution-semantics.md`: describe the wait next to the existing warm-reuse rules. ## Verification - Red on master: `pnpm --filter @paperclipai/server exec vitest run src/__tests__/environment-runtime.test.ts -t predecessor` fails both cases with `expected 'warm-2' to be 'warm-1'`: the successor created a second sandbox instead of resuming the first. - Green: the same command passes (3 tests). `environment-runtime.test.ts`, `plugin-warm-tasks.integration.test.ts` and `environment-run-orchestrator.test.ts` pass (288 tests). Server `tsc --noEmit` passes. - Not verified in a chaos wave. The window is a few seconds wide and appeared in 2 of 361 waves, so one wave cannot show the change. ## Risks - A successor can start up to 20 s later when its predecessor's release is slow. Creating a sandbox and restoring a backup takes longer, and failing needs a Retry. The two-minute bound keeps a stuck, never-released lease from delaying every later run. - If a caller held a lock that the predecessor's release needs, the wait ends at 20 s and acquisition behaves as before. - Compared with #20863 (fresh-sandbox fallback) and #20811 (unsaved provider state marker): this change does not alter either. It reduces how often a warm continuation reaches the fresh-sandbox branch. The backup restore proof is unchanged, so an unproven backup still fails closed with a working Retry. - The `workspace_finalize` deadlock in the same run is a separate defect and is not fixed here. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, file edit, test runs). Default context window, extended reasoning on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
6a43890326 |
Fix image attachments in remote native workspaces (#20939)
## Thinking Path > - Paperclip manages AI agents and their work. > - Native runs must receive the files that users attach to their messages. > - Remote wake preparation returned file metadata without readable bytes. > - The historical chat reader also rejected every remote target. > - Command-backed sandboxes and SSH targets already provide a bound command transport. > - This pull request stages verified files through that transport and clears them after the turn. > - Agents can then open attached images with their normal image reader. ## Linked Issues or Issue Description Refs #19740. This change covers SSH and command-backed sandbox execution, including Daytona. Plugin Runner tasks without a command transport still need a separate versioned file-transfer contract. This PR does not close that broader issue. The open attachment-recovery PR #20778 concerns provider-session attachment journals, not input files. **What happened?** A user uploaded an image and sent it to an agent. The remote native run received its filename and size, but no readable file. Wake preparation reported `remote_workspace_staging_unsupported`. The historical read fallback also refused remote execution. **Expected behavior** Copy the exact admitted attachment into the bound remote workspace. Let the provider read the image. Preserve company, task, source-comment, size, and hash checks. Clear temporary file contents when the turn ends. **Steps to reproduce** 1. Select a native agent on a command-backed sandbox or Linux SSH environment. 2. Attach a PNG to its current task or in-app chat message. 3. Ask the agent to inspect it. 4. Check that wake preparation supplies a readable workspace-relative image path. ## What Changed - Add a remote staging owner with bounded commands, stdin transfer, hash verification, opaque paths, and safe image extensions. - Confine Linux filesystem operations through directory descriptors. Refuse symlinks and replacement files during cleanup. - Bind current wake files and authorized historical external-chat reads to the selected lease transport. - Join active writes during cancellation. Retain failed cleanup work until scope close. - Tell providers to open image files with an image-capable reader. - Handle stdin stream errors in the shared SSH transport. Failed uploads return failure instead of raising an unhandled EPIPE or reporting success. - Add database, transport, cancellation, integrity, cleanup, executor-wiring, and Linux filesystem regressions. Document support and limits. ## Verification - Prepared an isolated source checkout with `node scripts/agent-preflight.mjs --skip-rust-debug`. - Focused suites: `pnpm exec vitest run server/src/services/native-runtime/remote-attachment-staging.test.ts server/src/services/native-runtime/native-runner-file-handoff.test.ts server/src/services/native-runtime/chat-attachment-read.test.ts` — 74 passed, 4 Linux-only tests skipped. - Executor checks: `pnpm exec vitest run server/src/services/native-runtime/native-session-executor.test.ts -t 'attachment staging to the remote|stages from the authenticated|verifies frozen staged admission'` — 7 passed. - Server typecheck: `pnpm --filter @paperclipai/server typecheck`, followed by the server layered check after the final changes. - `git diff --check` passed. The local secret, PII, and private-reference diff scan found no matches. - `pnpm verify` returned exit 2 because the documentation change requires broad CI. No full local suite is claimed. - The four Linux filesystem tests were skipped locally because this macOS host has no available Docker daemon. Linux CI executed all 17 remote-staging tests successfully, including those four confinement tests: [server shard 10](https://github.com/paperclipai/paperclip/actions/runs/43102035096/job/130796646266). - SSH regression: the real pipe test reproduced two unhandled EPIPE errors before the fix, plus false success when the peer exited with code zero. The fixed SSH tests and existing argv tests pass (12 tests). The remote-staging transport suite also passes (13 tests; 4 Linux tests require CI). - `pnpm --filter @paperclipai/adapter-utils typecheck` passed after the SSH change. - Final revision `1c5a9288b8d5f267ddeac84638ff6075dbbbf8ec`: all applicable CI jobs passed, including typecheck, build, general and serialized tests, Runner checks, canary dry run, and all eight browser shards. [CI run](https://github.com/paperclipai/paperclip/actions/runs/43102035096). Linux CI also passed all 6 native SSH tests: [server shard 5](https://github.com/paperclipai/paperclip/actions/runs/43102035096/job/130796646349). - Greptile reviewed that final revision at 5/5 with no remaining findings. The review thread is resolved. GitHub reports the PR mergeable with a clean merge state. - No live model turn or deployed release is claimed. ## Risks - Remote staging requires Linux, Node, and a command transport bound to the selected lease. Plugin Runner tasks without that transport retain their unavailable result. - Each file uses bounded independent commands. Slow transports can delay preparation or cleanup. File bytes stay out of command arguments and provider-facing error messages. - An abrupt controller or remote-host failure can bypass cleanup. Ephemeral sandbox disposal remains the backstop. Retained files are never blindly overwritten or deleted. - Previous-release provider sessions and checkpoints retain their format and identity. New turns receive newly staged input files. Existing restart-admission checks still apply. No migration, runner-image change, or account-credential transfer is required. ## Model Used OpenAI `gpt-6.1-sol` through the Codex harness in T3 Code, with xhigh reasoning, repository inspection, code execution, and tests. No subagents were used. The context-window size is not exposed by this runtime. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
757a20d2f1 |
fix(runner): keep remote recovery and launch through a short plugin RPC stall (#20945)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A remote run starts `paperclip-runnerd` in a sandbox. The controller
reads and controls it through the environment plugin
(`environmentExecute`).
> - #20940 made a stalled supervision probe "unknown" while the Runner
stays connected. Other callers still turn the same plugin stall into a
terminal failure.
> - During restart reattachment, the saved-state read, the
identity-marker read and the process check share a 30-second budget. One
plugin RPC that stalls for 40 seconds uses all of it, so a short stall
fails a reattachment to a healthy Runner.
> - At launch, a stalled launch call or identity read fails the run with
`runner_process_termination_unconfirmed` or
`runner_remote_process_identity_unavailable`, although the runner may be
healthy.
> - This pull request treats a plugin RPC timeout as "unknown" and
retries the read-only call within a bounded stall budget. A stalled
launch call is not repeated: the controller reads the launch nonce's
identity marker instead.
> - The benefit is that a short plugin stall during recovery or launch
no longer fails a healthy run. Every positive answer from the sandbox
keeps its meaning, and the bound still fails closed.
## Linked Issues or Issue Description
Refs #20940, #20909
**What happened?**
A plugin `environmentExecute` RPC stall of more than 40 seconds makes a
remote recovery read fail. Plugin RPCs time out at the requested timeout
plus a 30-second overhead, so a 10-second probe can take 40 seconds. The
reattachment read budget is 30 seconds. One stalled read therefore
failed reattachment with `probe_transport_failed` or `probe_timeout`. At
launch, a stalled launch call rejected the run with exit unconfirmed. A
stalled identity read (20-second wait, 2-second reads that can take 32
seconds) failed the run with
`runner_remote_process_identity_unavailable`.
**Expected behavior**
A plugin RPC timeout is a stall of the plugin, not an answer from the
sandbox. Read-only recovery reads retry within a bound. A launch call
that stalls is not repeated. The launch nonce's marker decides whether
the one runner was launched. At the bound, the run fails closed and
Retry works.
**Steps to reproduce**
1. Start a remote (Daytona) run.
2. Restart the controller so that the run reattaches, or start a new
run.
3. While the controller reads the sandbox, stop the plugin worker for 45
seconds.
4. The reattachment or launch fails.
**Paperclip version or commit**
master at
|
||
|
|
612cc7e3e6 |
fix(heartbeat): renew every legacy controller lease in one shared statement (#20924)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A legacy run holds a durable controller lease while it prepares. The server renews the lease every 10 seconds, and it stops the run if no renewal succeeds within the 60 second lease. > - Each run renewed in its own transaction on a private pool of two connections. Each renewal used six round trips (begin, statement timeout, transaction id, notify, update, commit) and woke live-run recovery for that run. > - When many runs prepare at the same time on a loaded database, the renewals queue behind each other. The queue grows with the number of runs, and every run loses its lease at about the same time. > - A healthy server must not fail its own runs because it has many of them. > - This pull request renews all leases of one server process through one shared queue. Requests that arrive while a renewal is in flight share the next statement. > - The benefit is that the renewal cost no longer grows with the number of live runs, so a burst cannot starve the lease. ## Linked Issues or Issue Description **What happened?** On a staging server that did not restart, 22 runs failed with `controller_lease_lost` in about 13 minutes. Every run was a legacy run in the `preparing` stage with `providerWorkStarted: false`. Every loss had cause `renewal_deadline`. The renewal had been pending 28 to 48 seconds (`renewalPendingMs`). The deadline callback ran on time (`deadlineLateMs` 3 to 861 ms), so the event loop was not blocked. The successful renewals before the loss had each taken 20 to 30 seconds, because the deadline fired only 25 to 48 seconds after the last renewal (`sinceRenewalMs`). About 30 runs were live in the process. The first wave started when a user retried about 14 blocked tasks at once. The automatic retries of the first wave then failed the same way, which made the second wave. 12 of 16 affected tasks became blocked. **Steps to reproduce** 1. Start about 30 legacy runs in one server process. 2. Make each renewal transaction slow (for example, a loaded database). 3. Wait one lease period. **Expected behavior** The leases stay owned. The renewal cost does not grow with the number of runs. **Actual behavior** All runs fail with `controller_lease_lost` (`renewal_deadline`). ## What Changed - `server/src/services/legacy-controller-lease.ts`: All controllers of one process renew through one queue per database. One flush is in flight at a time. Requests that arrive during a flush share the next flush, so a renewal waits behind at most one other statement. A flush is one transaction on the private lease pool: `statement_timeout` and one statement that renews all requested rows. - The statement locks the requested rows with `for update skip locked`. A row that another transaction holds is reported busy, and only that run retries after 250 ms. It does not stall the other runs in the batch. A row that is not this boot's live legacy lease is reported rejected, as before. - Renewal no longer emits the `active-execution-recovery` notification. A renewal only moves the deadline later. Recovery already has a deadline at the earlier expiry and rereads the moved deadline then. This removes one recovery pass, `pg_current_xact_id` and `pg_notify` per renewal. - The lease contract does not change: 60 second lease, 10 second renewal, boot-id fence, expired leases cannot renew, the local deadline is measured from the attempt start, and the loss causes and timings are the same. - `doc/run-log-events.md`: Documents the shared renewal queue. Upgrade story: no persisted shape changes. The row columns, lease length and boot-id fence are the same. A lease held by the previous release has a different boot id, so the new batch rejects it and leaves it untouched. A test covers a previous-release lease in the same batch. ## Verification - Red on `origin/master`, green with this change: - `keeps thirty preparing controllers owned when every renewal takes four seconds on a two-connection pool`: before, all 30 controllers lost the lease with `renewal_deadline` (the staging shape). After, none lose it. - `renews fifteen concurrent controllers in one statement` (real Postgres): before, 15 update statements. After, 1. - `renews through one private statement without waking live-run recovery`: before, the renewal woke `active-execution-recovery`. After, it does not. - `live-recovery-work.test.ts` `renews without waking recovery and re-reads the moved controller deadline at the old one`: before, 2 recovery passes per renewal. After, none, and recovery still moves to the new deadline at the old one. - Safety cases: a row lock on one run does not delay another run; a previous-release lease in the same batch stays untouched; a terminated lease connection is retried; existing deadline, retry, rejection and revocation tests pass. - `npx vitest run src/services/legacy-controller-lease.test.ts src/__tests__/live-recovery-work.test.ts`: 88 passed. - `pnpm verify`: see the CI checks on this head. ## Risks - Medium-low. All renewals of a process now share one statement. A failed flush fails every request in it; each watcher retries independently, as it did before. - A busy row is retried every 250 ms until the lease deadline. Before, the renewal waited on the row lock for up to the statement timeout. The loss behavior is the same. - Recovery wakes at the old lease expiry instead of after each renewal. It finds the live lease and moves to the new expiry. - Not in scope: the downstream workspace recovery that blocked tasks after a lost lease (`legacy_execution_requires_reconciliation`, `source_unavailable`). > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (claude-opus-5-5), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
228b0169be |
fix(dispatch): let the answer to the agent's own question continue the task it blocked on (#20931)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - An agent can ask the person a question (`request_human_input`) and wait for the answer > - A later comment from the person moves an open question into history (#15392). The agent still needs the answer, so it reports a blocker and the task becomes Blocked > - The person then answers. Dispatch cancels the answer's continuation with `issue_not_in_progress`, because only a dependency-blocked task may take an answer while Blocked > - The task rests Blocked with a failed run and a Retry. The answer is lost until a person clicks Retry > - This pull request lets the answer continue the task when the same agent blocked it to wait for that answer, proven from durable records > - The benefit is that the person's answer always reaches the agent that asked, without a Retry ## Linked Issues or Issue Description Refs #15392, Refs #20872 **What happened?** Restart-chaos waves with no fault injected fail every W2Q task on every provider (Claude, OpenCode, Codex) with `R3:retry_needed`. For Claude task t03 on master |
||
|
|
b01959cb15 |
fix(recovery): release a handed-off run's saved-files hold for the person it handed the task to (#20941)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - An agent can hand its own task to a person for review. With #20884, a native run that does this keeps running and finishes its turn. > - A restart can land between that hand-off and the end of the same turn. Restart recovery then settles the run as failed and keeps its unsaved sandbox files, so no work is lost. > - Automatic workspace recovery saves those files. But the workspace-repair hold then stayed on the person's task, because the hold only released for a newer agent owner. > - This pull request releases that hold for the person who received the task through the stopped run's own recorded hand-off, once the files are saved. > - The benefit is that the person's task rests in review with no stale "Saved work recovered" hold, and the agent's files are still saved first. ## Linked Issues or Issue Description Follow-up to #20884 (merged). It was reviewed there as a known limit: "Restart blocks the person's task". It uses the `resultJson.humanSelfHandoff` record that #20884 added. Related: #19184, which released the same hold for a newer agent owner. **What happened?** A native run handed its own task to a person, and a restart landed before the run finished its turn. Restart recovery settled the run and retained its files. After automatic recovery saved them, the task still had a workspace-repair hold ("Saved work recovered. Send a message to continue."). The person had nothing to do with that run's work, and nothing released the hold. **Expected behavior** The hold protects the workspace until the files are saved. After that, the task rests with the person and has no execution blocker. **Steps to reproduce** 1. A native run records a hand-off to a person, and the task is now `in_review` with that person. 2. A restart settles the run as failed, with its prepared remote workspace retained. 3. Automatic workspace recovery completes. 4. Before this fix, `getExecutionBlocker` still returns the run's workspace-repair hold. ## What Changed - `server/src/services/execution-blocker.ts`: `restoreHoldReleasedForCurrentOwnerPredicate` now also releases a recovered file-preservation hold in one more case. The task must be assigned to a person, and the stopped run's `resultJson.humanSelfHandoff` must name that task and that person. The existing proof is still required: the completed-recovery receipt bound to the action, and a copyback rather than a snapshot-only save. Until the files are saved, the hold stays. - `server/src/services/workspace-recovery-admission.test.ts`: two embedded-Postgres tests. - The hold blocks the person's task before recovery and releases after it. - A hand-off recorded for a different person keeps the hold. - `doc/execution-semantics.md`: one sentence on the restart case. ## Verification - Red: with the `execution-blocker.ts` change removed, `releases a handed-off run's recovered files hold for the person it handed the task to` fails. `getExecutionBlocker` still returns the hold after `recoverRetainedWorkspace`. - Green: `cd server && npx vitest run src/services/workspace-recovery-admission.test.ts src/__tests__/interrupted-preparation-hold-settle.test.ts src/services/explicit-native-continuation.test.ts src/services/native-runtime/native-human-self-handoff.test.ts`. All 617 tests pass. The server typecheck passes. - Chaos: the H2 workload from #20884 covers the hand-off with a K1 restart after it. Locally the runs had no unsaved files, so no hold appeared. This PR's case needs a remote sandbox with unsaved files. **No Daytona wave was run.** ## Risks - The release applies only when the stopped run's own record names the current person and task, and only after the completed-recovery receipt. Other holds, snapshot-only saves and unsafe archives are unchanged. - No schema change. A run without the record behaves as before. - This branch is rebased onto master after #20884 merged. The only conflict was an adjacent import line. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude (Anthropic), model ID `claude-opus-5-5`, through Claude Code with tool use and code execution. The session does not expose the context window or reasoning mode. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
48d23d1530 |
fix(runner): keep a connected remote Runner through a stalled plugin supervision probe (#20940)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A remote run starts `paperclip-runnerd` in a sandbox. The controller
monitors the process with a health probe that goes through the
environment plugin (`environmentExecute`).
> - A stall of the Daytona plugin worker makes that probe RPC time out
after 40 seconds. The monitor treated the timeout as "process
supervision failed".
> - The transport then failed closed with
`runner_process_termination_unconfirmed`. The run ended
`notification_transport_failed`, `recoverable: false`, and the close
path suspended a healthy Runner. The task rested blocked with Retry.
> - Nothing was wrong with the sandbox, the Runner or the provider. The
Runner held its own authenticated connection to the controller during
the full stall.
> - This pull request makes a failed or timed-out probe "unknown". While
the Runner stays connected on its own channel, the monitor keeps the
same runner and turn and probes again, for a bounded time.
> - The benefit is that a short plugin stall no longer kills a healthy
run. The fail-closed cases stay the same.
## Linked Issues or Issue Description
Refs #20909 (chaos cell CR3, which reproduces this failure)
**What happened?**
A 45-second stall of the Daytona plugin worker killed a healthy run.
Chaos cell CR3 reproduced it 2 of 2 times (OpenCode and Claude,
Daytona). A staging run had the same failure with no deploy: a plugin
`environmentExecute` RPC stalled for more than 40 seconds. The run
failed with "Notification transport failed closed:
runner_process_termination_unconfirmed: runnerd process supervision
failed: RPC call \"environmentExecute\" timed out after 40000ms". The
close path then sent `runner.suspend`, and the healthy Runner suspended
and exited. The task rested blocked with Retry.
**Expected behavior**
A probe that times out or fails in transport is not proof that the
process stopped. While the Runner stays connected on its own channel,
the run continues. If supervision cannot come back within a bound, the
run rests with a working Retry. A probe that positively reports the
process gone still starts the existing recovery.
**Steps to reproduce**
1. Start a remote (Daytona) run with a T1 workload.
2. Send SIGSTOP to the server's Daytona plugin worker for 45 seconds,
then SIGCONT.
3. The run fails `notification_transport_failed` about 40 seconds into
the stall.
**Paperclip version or commit**
master at
|
||
|
|
edd00526e7 |
fix(native): park a response_wake with no admitted response instead of self-waking (#20935)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A native agent run ends with a structured result. The status arbiter
turns that result into an issue status and next actions.
> - A result can report `yielded` with `continuation.kind:
"response_wake"`. This means "wake me when somebody responds".
> - Only a direct Board comment gave this kind a passive wait. Every
other `response_wake` became an immediate `issue_status_changed` wake to
the same agent.
> - The agent woke, saw nothing new, reported `response_wake` again, and
woke again. Nothing bounded this loop. It spent provider budget and its
back-to-back runs caused other failures on warm sandboxes.
> - This pull request parks such a result as a visible wait, and puts
declared continuation wakes under the rewake throttle as a backstop.
> - The benefit is that an agent that waits for input stops and costs
nothing until a real response arrives.
## Linked Issues or Issue Description
**What happened?**
Restart-chaos waves on master `683fd1e9a5` (no faults injected) showed
one task run 30 to 66 times, about 12 s apart. Example: wave
`w-20261111T190352-s5101`, task t02 (W2C), 65 runs in 13 minutes. Each
run reported `yielded` with `continuation.kind: "response_wake"` and the
summary "No changes made. This status-change wake does not trigger phase
2". Each run was woken by `issue_status_changed`, and each finalization
committed `live_continuation_registered` with an `enqueue_continuation`
effect of kind `response_wake`.
Root cause: `arbitrateNativeStatus()`
(`server/src/services/native-runtime/status-arbiter.ts`) only treats
`response_wake` as passive when `boardResponseWaitAuthorized` (from
`readNativeBoardResponseWaitSource`), external chat, or plan wait is
true. Otherwise it falls to the generic yielded branch, and
`status-decision-committer.ts` (`enqueue_continuation`) enqueues an
`issue_status_changed` wake to the same agent. `issue_status_changed` is
not in `THROTTLED_ISSUE_REWAKE_REASONS`, and the native committer writes
an `issue.updated` activity row with the run's id for every decision, so
the rewake throttle saw every native run as progress.
**Expected behavior**
A `response_wake` with no admitted response source parks the task. A
comment, an interaction answer or a status change by somebody else wakes
the agent. No path can loop without a bound.
**Steps to reproduce**
Run `tests/restart-chaos/main.ts --env local --adapters codex
--workloads W2,W2Q,W2C --concurrency 10 --tasks 10 --seed 5101
--lifecycle warm --triggers time`. W2/W2C tasks loop after phase 1.
## What Changed
- `status-arbiter.ts`: a `yielded` / `response_wake` result with no
admitted response source and no blocking remaining work now commits
`response_wake_waiting_for_input`, keeps the task `in_progress`, and has
no effects. Policy version is `phase6-v14`. Board, external chat, plan,
governance, dependency, pause and unfinished-work branches are unchanged
and still come first.
- `native-runtime/native-response-wake-wait.ts` (new) and
`recovery/service.ts`: recovery treats that applied decision as a
durable wait while it is the issue's latest decision, the issue is
`in_progress` with the same agent assignee, and no user comment arrived
after the run started. This applies to the stranded-work sweep and to
the paused-agent passive-wait check.
- `execution-projection.ts`: a committed run with this reason shows the
phase `waiting_for_answer`, label "Waiting for a reply", and a next
action that tells the user to comment or change the status. This shows
only while that decision is still the issue's latest decision and the
issue is `in_progress`; an older run shows "Completed".
- Backstop in `issue-rewake-throttle.ts` and `heartbeat/queue.ts`: an
`issue_status_changed` wake that carries a declared `continuationKind`
(the `enqueue_continuation` effect) is now a throttle candidate. The
native status committer's own `issue.updated` / status-decision activity
no longer counts as run progress or as new input. After two consecutive
successful runs with no issue-visible progress, such a wake waits for
the existing escalating cooldown (2 min, doubling, max 30 min). The
native status dispatcher (`dispatchNativeStatusWakeBatch`) does not drop
a throttled intent: it keeps the intent `queued`, reports the cooldown
deadline to the deferred-receipt worker, and dispatches it again when
the cooldown ends. Status wakes without a declared continuation (blocker
owner, notify owner, retry) are unchanged.
- Restart chaos: new enforced universal check U5 (`universal.ts`). A
task with more than 10 runs fails as `U5:run_budget_exceeded`
(`RESTART_CHAOS_U5_MAX_RUNS`). The README records the calibration.
- `doc/architecture/native-status-arbitration.md`: the decision table
and the wait rules.
## Verification
- RED on master (same reason as the waves):
`heartbeat-process-recovery.test.ts` "parks a native response_wake that
no response admitted…" fails with decision `reasonCode:
"live_continuation_registered"` and effect `{ kind:
"enqueue_continuation", continuationKind: "response_wake" }`.
`heartbeat-issue-rewake-throttle.test.ts` "throttles a
same_agent/response_wake/retry continuation…" fails 3/3 because the wake
is admitted.
- GREEN: the integration test finalizes the run, dispatches native
status wakes and runs the stranded-work sweep three times. It asserts
one run total, one wake request, no recovery action, the task
`in_progress`, and the projection "Waiting for a reply".
- `heartbeat-process-recovery.test.ts` "delays a no-progress native
continuation until the rewake cooldown…" finalizes a real `same_agent`
continuation after a no-progress run, dispatches twice (one throttle
receipt, intent stays `queued`, deadline equals `nextAllowedAt`), then
dispatches after the cooldown and gets the run. It fails when the
dispatcher drops the throttled intent.
- Focused suites pass: `heartbeat-process-recovery`,
`heartbeat-issue-rewake-throttle`, `status-arbiter`,
`native-status-arbiter-corpus`, `runtime-mode`, `execution-projection`,
`chat-run-milestone-work`, `heartbeat-dependency-scheduling`,
`heartbeat-run-summary` (904 tests), and all restart-chaos unit tests
(584 tests). `pnpm --filter @paperclipai/server typecheck` passes.
- Universal audit over the 17 saved waves of 2026-11-11: U5 flags 12
tasks, all looping tasks (12 to 66 runs), and 0 tasks on all-pass waves.
- Chaos rerun of local warm seed 5101 (W2, W2Q, W2C): result is added in
a comment on this PR.
## Risks
- Behavioral shift: a model that reports `response_wake` to mean
"continue now" no longer gets an immediate run. The task waits visibly
for a reply. The runner prompt already describes `response_wake` as a
real response wait, and `same_agent` stays available for server-owned
continuations.
- The throttle now applies to native runs. A chain of declared
continuations with no issue-visible output (no comment, document, work
product or other activity) waits for the cooldown after the second such
run. `same_agent` continuations are server-owned (repository
preparation), so this affects only repeated no-output chains.
- Old saved state: decisions committed under `phase6-v13` keep their
recorded effects. A run that finalizes after deploy uses the new policy.
No migration.
- Budget hard-stop behavior is unchanged.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test runs).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
b2779054d4 |
fix(issues): let a native run hand its own task to a person without cancelling itself (#20884)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A task has one assignee. An agent often finishes its work by handing its own task to a person for review. > - The task update route stops the outgoing run before it saves a reassignment. > - Since #16082, a legacy run that hands its own task to a person is exempt, so it finishes its turn. Native runs were not exempt. > - On staging, a native run (ACPX Claude, 11.5 minutes of work) sent `PATCH /api/issues/{id}` to hand its task to a person. The route cancelled that run with `issue_reassigned`. The agent saw `api_transport_failure`. The task moved to the person, but it kept a stale execution blocker ("Saved work recovered. Send a message to continue."). > - The native runtime assumes a running run still owns its task. So widening the exemption alone is not enough: the run then fails after its result is accepted, and its finalizer can act on the person's task. > - This pull request applies the exemption to native runs and makes the native terminal fence, restart recovery and finalizer handle a run that gave its task away. > - The benefit is that an agent can hand its task to a person and finish normally, with no blocker on the person's task. ## Linked Issues or Issue Description Refs #16082 (legacy exemption). Related: #20541 (agent-to-agent self handoff) and #18293 (agent key handoff). Those PRs cover other actors and targets. This PR covers native runs that hand their task to a person. **What happened?** A native agent run handed its own task to a person (`assigneeAgentId: null`, `assigneeUserId`, status `in_review`) through its signed run token. The route cancelled the run that sent the request (`issue_reassigned`, "Cancelled before issue reassignment"). The cancelled native run then went through workspace recovery. The person's task kept an execution blocker with `restore_failed`. **Expected behavior** The run that hands its own task to a person keeps running and finishes its turn. The task rests `in_review` with the person. No execution blocker or recovery action stays on the task. The outgoing run cannot change the task after the hand-off. **Steps to reproduce** 1. Run a native agent on its assigned task. 2. From that run, send `PATCH /api/issues/{id}` with `assigneeAgentId: null`, a person's `assigneeUserId`, and status `in_review`. 3. Before this fix, the route cancels the run with `issue_reassigned`. **Why the exemption was legacy-only** #16082 limited the exemption to the run shape of its incident and kept a verified stop for every other case. The new H2 chaos workload showed why native needed more than the route change. With only the route widened: 1. The run's terminal fence `assertCurrentWakeCommentsRead` requires the run to still own the task. It threw `native_current_wake_comments_binding_changed` after the result was accepted. The run then failed with `native_session_lease_lost`, and its finalizer recorded `workspace_finalization_failed` as an agent-owned recovery on the person's task. 2. A restart after the hand-off left the run `running`, because restart recovery refused it with `issue_execution_lock_changed`. 3. The native finalizer only superseded a `done` decision. A `needs_review` report moved the person's task back to `in_progress` and queued a continuation for the outgoing agent. 4. The REST guard against the outgoing run's later mutations (`issue_handed_off`) applied to legacy runs only. ## What Changed - `server/src/routes/issues.ts`: the signed current-run human self-handoff exemption now applies to native runs. All other checks stay: exact run, agent, company and task, `agent_jwt` source, human recipient. For a native run the route first records the hand-off on the run (`resultJson.humanSelfHandoff`, new `native-human-self-handoff.ts`). Board reassignment, agent keys, and handoffs to an agent or to nobody still stop the outgoing run. - `current-wake-comments.ts`: the terminal fence accepts a run whose recorded hand-off matches the person who now owns the task. Without the record (for example a board reassignment), or after a later reassignment, it still refuses. - `native-restart-recovery.ts`: after a restart, a handed-off run is not resumed. It passes the execution-lock check and is then settled by the existing ownership gate (`task_ownership_or_status_changed`) without changing the person's task, instead of staying `running`. - `status-arbiter.ts` and `native-run-finalizer.ts`: `supersedeHandedOffDecision` keeps the person's task status and strips retries, continuations, owner wakes, recoveries, blockers and reviewer bindings, also from decisions that already preserve the status. Two effects stay: a governed request's interaction, and the workspace-finalization repair, so saved work is not lost. Review runs are not affected. - `server/src/run-task-ownership.ts` (renamed from `legacy-run-task-ownership.ts`): `assertHandedOffRunTaskOwnership` rejects later task mutations from the outgoing legacy or native run, in the route and under the update transaction's locks. - `tests/restart-chaos/`: new opt-in workload `H2`. The agent saves a brief, then hands the task to its creator in review as its last step. The oracle requires the task to rest `in_review` with the creator, every run to settle without `issue_reassigned`, readable non-empty run history, and no execution blocker or active recovery action. - Docs: `doc/execution-semantics.md`, task feature map and agent access map, chaos README. ## Verification - Red on `origin/master` (same tests, master source): the route cancels the native run (500, and 409 `runner_goal_reassignment_stop_unconfirmed`); the finalizer moves the person's task to `in_progress` with a continuation for the outgoing agent; a late `done` from the outgoing native run closes the person's task. - Red chaos wave on master's server (`--server-from` a master checkout, `--adapters claude --workloads H2 --faults K1 --matrix`): 0/2 pass. Both runs ended `cancelled` / `issue_reassigned` (`T:handoff_did_not_cancel_run`). - Green focused tests on this branch: - `native-human-self-handoff.test.ts` (embedded Postgres): hand-off through the real issue service; the run stays `running`; `done`, `needs_review` and failed-turn reports leave the task `in_review` with the person, with `getExecutionBlocker` null and no recovery action; the terminal fence accepts only the recorded hand-off; a late change from the outgoing run fails with `issue_handed_off`; pure policy cases. - `issue-agent-mutation-ownership-routes.test.ts`: native and legacy self-handoffs persist without `cancelRun`, and native records the hand-off before the update. Board, agent key, other run, task, agent and company still stop the run. - Native runtime suite (restart recovery, wake comments, arbiter, stale-owner completion), `issues-service`, `low-trust-red-team-routes`: pass. Server and restart-chaos typecheck pass. `node scripts/check-maps.mjs` passes. - Green chaos waves on this branch (local, `--adapters claude --workloads H2`): - `--matrix --faults K1 --seed 4459`: 2/2 pass (K0 baseline and a K1 crossing). - `--tasks 3 --faults K1 --fault-after-seconds 60` and `--fault-after-seconds 100`: 3/3 pass each. In the first, all three restarts landed after the hand-off while the runs were live; the runs settled and the tasks stayed with the person. - The final waves ran on the pre-rebase head `8a2a3b6a74`. The rebase onto master only merged the chaos workload lists (`types.ts`, `config.ts`, `journeys.ts`). No Daytona wave was run. ## Risks - The outgoing native turn keeps running after the person owns the task. Its native task tools already require the run to own the task, so they refuse (`paperclip_runner_tool_binding_not_authorized`). Its REST mutations are rejected with `issue_handed_off`. - A restart after the hand-off settles the run as failed (`native_restart_recovery_blocked`) instead of resuming it. The person's task is not changed and needs no Retry. - The finalizer rule covers any native run whose task belongs to a person at finalization, not only a self-handoff. Before this change such a run could still move the person's task and queue its own continuation. - Saved state: `resultJson.humanSelfHandoff` is a new optional run key. An older server ignores it, so a run handed off on the new server and finished on an old one fails its fence as today. No schema change. - No API shape change. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude (Anthropic), model ID `claude-opus-5-5`, through Claude Code with tool use and code execution. The session does not expose the context window or reasoning mode. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |