mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-09 16:02:36 +01:00
4dc5281f5190cf5a4f44ef11cfcbaa210aff1784
334
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 DottaandClaude Opus 5.5
|
fc7e3a6503 |
fix(recovery): never replace a native session after a deterministic refusal (#19733)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native agent runs keep a saved provider session, and same-run
recovery resumes it after a failure
> - When resume fails and the checkpoint allows it, recovery archives
the saved continuity state and opens a fresh replacement session
> - Recovery used that path for every resume failure, including
deterministic refusals that left the saved state unchanged and repeat on
every retry
> - In five staging tasks a refusal (a protected launch argument, or a
predecessor tool call with no result) escalated into a replacement. The
remote archive move failed part way, the next attempt found no runner
state, and each task was blocked for workspace repair
> - This pull request stops before replacement after a closed-class
refusal, keeps the saved state, and hands the run to an operator with a
named reason
> - The benefit is that a deterministic refusal can no longer damage
saved state or open a replacement over an unsettled external action
## Linked Issues or Issue Description
Refs #19659, Refs #19516, Refs #19334
**What happened?**
Five Codex tasks on Daytona sandboxes failed on staging. The first
attempt failed with a deterministic refusal:
- `run.attach cannot change protected launch arguments` (four tasks,
fixed separately in #19659), or
- `native_runner_source_semantic_reconciliation_required` with a
predecessor `paperclip_call_assigned_tool` call still pending (one
task).
The same-run retry then opened a replacement session.
`archiveContinuityState` failed with
`runner_continuity_break_archive_failed` and no detail. The third
attempt failed with `runner_remote_state_unavailable: exit=1`, and the
task was blocked with "Workspace repair required".
**Steps to reproduce**
1. Save a native session whose checkpoint policy is
`allow_replacement_after_resume_failure`.
2. Make the resume fail with a refusal that does not change state (an
attach refusal, or a pending tool call).
3. Same-run recovery opens a replacement session and archives the saved
continuity state.
**Expected behavior**
A deterministic refusal stops the run for an operator. The saved state
stays intact. A run with a pending side-effecting call never reaches
replacement. Transport drops keep the existing retry behavior.
**Root cause**
`executeNativeSession` checked only the checkpoint's replacement policy.
The closed failure class (#19334) only had an effect when replacement
was forbidden. Attach refusals and pending tool calls had no class at
all, and the source-reconciliation refusal was a plain `Error`.
## What Changed
- Runner (`provider_backend.rs`, `durable/mod.rs`): `run.attach`
refusals that would change the durable Codex profile or a protected
launch argument carry the closed classification
`provider_attach_refused`. An attach blocked by
`durable_pending_tool_calls` carries `provider_side_effect_pending`.
These refusals already leave the saved state unchanged.
- Contracts (`harness-driver.ts`): new closed classes
`PROVIDER_ATTACH_REFUSED` and `PROVIDER_SIDE_EFFECT_PENDING`. A typed
`NativeRunnerSourceReconciliationRequiredError` replaces the plain
`Error` at the five `native_runner_source_*_reconciliation_required`
throw sites. `runnerRecoveryFailureClass` reads the authenticated
command-result classification or the typed error, never message text.
- Runtime (`native-session-runtime.ts`): after either class, recovery
never opens a replacement session, whatever the checkpoint policy.
- An attach refusal ends with the existing
`NativeProviderStateIncompatibleError`, which maps to the existing
operator path `native_provider_state_incompatible`.
- A pending side effect ends with the new
`NativeProviderSideEffectPendingError`.
- Server (`native-session-executor.ts`, `native-session-resume.ts`,
shared `EXECUTION_RECONCILIATION_CAUSES`):
`native_provider_side_effect_pending` is a permanent failure and an
execution-reconciliation cause, so the board gets a reconciliation
(record whether the action ran) instead of a generic Retry. The
reconciled continuation does not rebind the saved session, which still
refuses attachment. It starts a fresh provider session in the same
workspace. The saved session and workspace are unchanged.
- Archive (`archiveContinuityState`): a failed remote move now records
the bounded (512 characters), redacted exit status and output in
`runner_continuity_break_archive_failed: exit=… timedOut=… <output>`.
The remote move already ran before the local move on `master`, so a
failed remote move leaves the local state in place.
`restart-interrupted-turn.ts` accepts the detailed message for its
reattachment-loss retry.
- Docs: `doc/execution-semantics.md` and
`packages/paperclip-runner/docs/durable-recovery.md` describe the new
rule.
## Verification
Red before the fix, green after (each test first run against unchanged
source from `origin/master`
|
||
|
DottaandClaude Opus 5.5
|
c45c7cb854 |
fix(workspaces): ignore killed-transfer scratch in sparse checkpoints (#19730)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native run in a remote sandbox copies its workspace back to the host after the turn. A sparse checkpoint copies only changed entries, and the host validates each entry against the sandbox's own manifest > - Every Daytona transfer stages a `.paperclip-upload-<uuid>` scratch file at the sandbox workspace root and removes it only in the same process > - A control-plane kill during any transfer leaves that scratch behind. The sparse checkpoint then recorded it as a workspace file, but providers never export it. Every later copyback and every saved-work recovery attempt failed with "Workspace checkpoint payload mismatch" > - The failed recovery then left an accepted Retry showing "Paperclip is finishing the saved result" with no run > - This pull request stops recording transfer scratch, makes the checkpoint export an exact copy, names the failing entry, and gives a Retry behind a spent recovery an honest reason and a working action > - The benefit is that a restart during a transfer no longer makes a task's saved work permanently unrecoverable ## Linked Issues or Issue Description Refs #19721 (RC-50: the killed copyback owner; this PR covers the separate RC-50b cause) **What happened?** Restart-chaos waves s12/t01 and s14/t05 (Claude on Daytona). After a SIGKILL of the control plane, the native result was accepted, and then workspace copyback failed with "Workspace checkpoint payload mismatch". In s14 all 6 saved-work recovery attempts failed the same way. A Retry accepted during recovery kept `retryWait.reason = finalizing_saved_result` after the recovery budget was spent. No run was created. **Steps to reproduce** 1. Start a native Claude run in a Daytona sandbox. 2. SIGKILL the control plane while any provider transfer is in flight. 3. Let the run finish its turn and copy its workspace back. **Expected behavior** Copyback succeeds. Provider scratch is not workspace content. A Retry behind a spent recovery names the failure and can start a new recovery. **Actual behavior** The s14 sandbox (6f4a0b6a) still held `/home/daytona/paperclip-workspace/.paperclip-upload-d99fe36b-5acc-437b-96ad-85d5ff9e79c4.tar`. Its mtime was 03:29:23, the second of the K4 SIGKILL. The Daytona provider stages transfer scratch at `remoteDir`, which is the lease `remoteCwd` (the workspace root). The runtime workspace wipe deliberately preserves `.paperclip-upload-*`. The checkpoint helper (`workspace-checkpoint.ts`, `checkpointMain`) recorded the stale tar. The ordinary export excludes `.paperclip-upload*`, so the host received nothing for that entry. `validateCheckpointPayload` then threw "payload mismatch" on every attempt. The file-mode hypothesis was disproved: the sandbox umask is 022 and the scratch is mode 644. ## What Changed - `packages/adapter-utils/src/exclude-patterns.ts`: new `isSandboxTransferScratchPath`, which matches a root-level `.paperclip-upload-*` entry. This agrees with the wipe's preserve rule in `buildWorkspaceTarExtractCommand`. - `packages/adapter-utils/src/workspace-checkpoint.ts`: the sandbox checkpoint helper skips that scratch. A mismatch error now names the entry and its recorded, baseline and received forms (kind, octal mode, short hash; never contents). - `packages/adapter-utils/src/sandbox-managed-runtime.ts`: the checkpoint export requests `preserveSourceEntries: true`. The checkpoint output is an exact copy that the host validates entry by entry. An ordinary export applies the host umask and drops `._*` and `.paperclip-upload*` names at any depth. That gives the same permanent mismatch for a legitimate user file such as `._summary`, `notes/.paperclip-upload-plan.md`, or a `0775` script. - `server/src/services/explicit-native-continuation.ts` and `task-interruption-presentation.ts`: when the run's saved-work recovery is exhausted, a waiting Retry reports `saved_work_recovery_failed` with Retry available, instead of `finalizing_saved_result`. The existing task Retry path starts one new recovery generation. - `doc/execution-semantics.md`: a short additive note. ## Verification - `packages/adapter-utils/src/workspace-checkpoint-capture.test.ts` uses the real checkpoint helper and an export that behaves like Daytona's ordinary export: - Before the fix: `Workspace checkpoint payload mismatch at ".paperclip-upload-d99fe36b-….tar": sandbox recorded file mode=644 …; baseline absent; received absent`. That is the wave error. - After the fix: the scratch is not recorded, and validation passes. - A second case shows that reserved-looking user names and `0775` modes need the exact export. - `packages/adapter-utils/src/workspace-checkpoint.test.ts` is an end-to-end `prepareSandboxManagedRuntime` restore through a provider model. In that model, ordinary exports drop reserved names and apply the umask, and preserve exports copy exactly. Two red checks: - Without `preserveSourceEntries`, it fails with the mismatch on `._summary`. - Without the scratch skip, the stale tar leaks into the host workspace. - Green: the agent file, the `._summary` file and the `0775` mode arrive, and no scratch arrives. - `server/src/services/workspace-recovery-preparation-gates.test.ts`, "names a spent saved-work recovery behind an accepted Retry, and the next Retry finishes it" (s14 shape): - Red: `finalizing_saved_result` after 6 failed attempts. - Green: `saved_work_recovery_failed` with `canRetry: true`. The next Retry starts a new generation, recovery completes, and the Retry continues in exactly one new run. - Suites: adapter-utils checkpoint, capture, sandbox-managed-runtime, seed-reuse and managed-seed (168 passed). Server preparation-gates (47), explicit-native-continuation and task-interruption-presentation (455), workspace-recovery admission and retry-copyback, and native-workspace-finalizer. `tsc --noEmit` is clean for adapter-utils and server. - Live evidence: I started the stopped s14 sandbox read-only, listed its workspace, and stopped it again. No Daytona replay was run. The runner content ID of this checkout matches no built image. ## Risks - A user file named `.paperclip-upload-*` directly at the workspace root is no longer captured by a sparse checkpoint. Providers already drop that name from every ordinary export, and the runtime already reserves it at the root. - The checkpoint export now uses the provider's exact-copy mode. A workspace file with setuid, setgid or sticky bits is refused by that mode, as it is for the existing exact-copy transfers. The ordinary export would have stripped those bits and failed validation anyway. - The stale scratch tar stays in the sandbox. It is ignored, not deleted, because a concurrent transfer can own a file with the same prefix. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, file edits, test runs, embedded Postgres, a read-only Daytona sandbox listing). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
5cc89b1e4d |
fix(native): do not charge a restart resume that made progress before the next crash (#19720)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs have an execution recovery budget of three provider attempts, so a run that keeps failing stops retrying > - A control-plane crash (no graceful drain) makes the next server resume the run, and that resume charges one attempt > - #19625 stopped charging a resume after a graceful drain, but crash restarts still charge, even when the previous resume did real work > - On a deploy day with several restarts, a healthy run used up its budget and the task needed a manual Retry > - This pull request does not charge a crash resume when the previous restart resume made recorded progress > - The benefit is that restarts alone do not exhaust a run that keeps making progress, and genuine loops stay bounded ## Linked Issues or Issue Description Refs #19625 **What happened?** In restart-chaos testing (storm wave, OpenCode run), a run crossed four control-plane restarts: K1, K1, K2 (drain), K1. Each resume was clean: the runner restored the session, recorded `provider_turn_lost_on_restore`, and the #19699 continuation turn then ran tools for minutes. The two K1 resumes each charged one attempt (attempt 1 to 3). The drain resume did not charge (#19625). The fourth restart found attempt 3 and blocked the run with `native_restart_recovery_blocked` / `execution_recovery_budget_exhausted`. Seven runs in the same wave failed in the same way. In every case, each resume had accepted a new turn and completed tool work before the next restart. **Expected behavior** A run must not fail because it survived several controller restarts that each resumed cleanly and made progress. The budget must still stop a resume that fails each time, a crash loop that makes no progress, and a provider that fails repeatedly. **Steps to reproduce** Start a native local run. Kill the server (SIGKILL) while the run is active and let it resume. Let the resumed turn run tools. Repeat three times. The third crash resume is blocked with `execution_recovery_budget_exhausted`. **Paperclip version or commit** `master` at `8a34c1b62c`. ## What Changed - `claimNativeRestartRecoveries()` (`native-restart-recovery.ts`) records `progressWatermarkSeq` (the run's highest `heartbeat_run_events.seq`) in the history entry of each recovery claim. - New `previousResumeMadeProgress()`: the next claim reads the watermark of the most recent claim. It finds progress when, after that watermark, a `turn.accepted` event names a turn, and a later `tool.execution.completed` or `item.completed` event (not a `usage` item) belongs to the same turn. The evidence is committed run events, not timing. - When that progress exists, the claim is not charged (`chargesProviderAttempt: false`, and the `>= 3` exhausted gate is skipped). This uses the same path as the clean-drain rule from #19625. The history entry records `attemptKeptForProgress: true`. - Restore activity does not count: lineage items without a turn, `turn.failed`, work on the interrupted turn, `turn.submitted`, and an accepted turn with only usage or a failure. So a resume that dies or fails before it does work still charges. - The credit stops at controller generation 12 (`NATIVE_PROGRESS_CREDIT_GENERATION_LIMIT`). A run that does some work and then crashes the controller every time still reaches the budget. - The first restart (which interrupts the original execution, not a resume) still charges. A history entry from an older release has no watermark, so it gets no credit. - `doc/execution-semantics.md` states the rule next to the three-attempt budget. ## Verification New tests in `server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts` (embedded Postgres, real `claimNativeRestartRecoveries`; between claims the test applies the executor's attempt charge and marks the controller dead): - `keeps recovering a run whose restart resumes each completed work before the next crash`: five crash resumes, each followed by restore noise and one completed tool in the continuation turn. Charges are `[true, false, false, false, false]` and the attempt stays 2. Red on master (attempt 2, 3, then blocked). - `still exhausts the budget when restart resumes complete no work between crashes`: resumes followed only by restore noise and a failed continuation turn. Attempt 2, attempt 3, then `execution_recovery_budget_exhausted`. Green on master and with the fix. - `bounds a crash loop that makes progress before every crash`: credit stops at the generation limit, the last attempt is spent, and the run blocks. Red on master. - `native-runner-restart-recovery.integration.test.ts`, `heartbeat-native-restart-suspension.test.ts`, `native-restart-recovery-isolation.test.ts` and `native-remote-retry-admission.test.ts`: 218 passed, 14 skipped. - I ran the progress query against the live database of the failing wave. For all seven blocked runs, every resume had progress before the next restart, so this rule would have kept each of them recovering. - `tsc --noEmit` for `server` has no errors in the changed files. ## Risks - A run that makes progress and then makes the controller crash each time gets up to about ten extra resumes before the budget applies. The generation limit bounds this. - The rule depends on the PRP event shape (`payload.prpEvent.turnId` on `turn.accepted` and on completed work). Events without that shape get no credit, so the failure mode is the old behavior (charge), not an unbounded loop. - The query runs once per claim, on the `(run_id, seq)` unique index range after the watermark. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test execution). Extended reasoning enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
51eb4ff705 |
fix(recovery): finalize an accepted native result after a hard kill (#19713)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native run hands its accepted result to a controller lease, and the controller finalizes the task from that result > - A graceful shutdown releases that lease for the next server (#19591), but a hard kill (SIGKILL, OOM, host crash) cannot > - After a hard kill between result acceptance and finalization, the next server waited for the dead controller's 20-minute lease, so the run stayed `running` and the user's follow-up comment waited with it > - This pull request releases that lease at startup when the controller is proved dead by its boot ID and process identity, never by age > - The benefit is that the saved result finalizes within seconds after a restart, without another provider turn, and queued follow-up work continues ## Linked Issues or Issue Description Refs #19591 (the graceful-shutdown counterpart) and #19578 (boot-identity proof for legacy runs). **What happened?** A restart-chaos wave sent K1 (SIGKILL) 20 ms after a Claude run recorded `run.result.accepted` and `run.terminal`. The run then stayed `running` for more than 10 minutes, through four more restarts. The finalization row was `workspace_finalizing` with the stored result. Its lease belonged to the killed server: a different `controller_boot_id`, a `controller_pid` with no process, and `lease_expires_at` 20 minutes ahead. Startup recovery skips runs that already have a result. The result-commit worker waits until the lease expires. The task's follow-up comment could not run until then. **Steps to reproduce** 1. Let a native local run accept its result. 2. SIGKILL the server before finalization finishes. 3. Restart the server. The run stays `running` until the old lease expires. **Expected behavior** After a restart, a result lease held by a stopped controller of an earlier boot is treated as released at once. The stored result finalizes once, with no new provider turn, and the follow-up wake runs after it. ## What Changed - New `server/src/services/native-runtime/native-dead-controller-result-lease.ts`: `releaseDeadControllerResultLeases` finds `running` native runs with an accepted result and an unexpired lease. It requires the lease to belong to a different controller boot. It proves the controller stopped with `evaluateNativeControllerTakeover` (`requireStoppedController`): the process is dead, or its PID now names another process. It then expires that lease with a compare-and-set on the exact owner token, boot ID, and PID, and notifies the result-commit and finalization workers. - `server/src/services/heartbeat/recovery.ts`: `recoverNativeRunsAfterRestart` calls this first and records a `native.recovery.transition` event (`awaiting_result_finalization`) for each released run. - Only executor leases are released, because only the executor claim writes the controller identity together with the lease. Finalizer (`native-finalizer:`) and cleanup (`native-cleanup:`) claims keep an earlier executor's identity, so they are never released this way and expire on their own short leases. A finalization-resume token (`:resume:`) on a result row always belongs to the executor that adopted it, because a pending resume claim is taken only on a row without a result. - Also not released: a live owner, an owner without a recorded process, a lease of the current boot, a run this process is executing, and remote workspaces (`nativeWorkspaceSync` or `remoteWorkspace`), which keep their fenced lease. - The compare-and-set does not compare the stored expiry. Postgres stores microseconds that a JavaScript `Date` drops, and an exact-expiry comparison silently skipped the real lease in the live reproduction. - `doc/execution-semantics.md`: documents the rule. ## Verification - Red then green on embedded Postgres (`server/src/__tests__/heartbeat-native-restart-suspension.test.ts`): - A hard kill with a dead controller's executor lease (first claim, restart claim, or resumed claim) now finalizes on the next startup: run `succeeded`, finalization `committed`, task `done`, one result, one run, no provider dispatch. These tests failed before the fix with `status: running`. - A lease expiry stored with microsecond precision is released. This test failed against the first version of the fix and reproduces the live miss. - A live controller of another boot keeps its lease, and so does a lease of the current boot. - A live claimant's finalizer or cleanup lease is kept when the recorded earlier executor is dead. - `npx vitest run src/__tests__/heartbeat-native-restart-suspension.test.ts`: 20 passed. - Related suites pass: `native-session-resumption`, `native-runner-restart-recovery.integration`, `native-run-projection-work`, `live-recovery-work`. One `live-recovery-work` legacy case timed out once under load and passed when run again by itself. - Live, local: `tests/restart-chaos/main.ts --workloads C1 --triggers result --faults K1 --rc06-masked`. The failing case above came from such a wave. I started its kept database and confirmed the stored lease state and the compare-and-set miss. Against that same database, the fixed release expires the stuck executor lease. Two later waves (12 tasks, `--triggers result`) passed every task. Run events were not kept for passing tasks, so I cannot confirm that a kill landed in the narrow pre-finalization window in those waves. - Live proof on head `8971874654`: local ACPX Claude, workload A1, seed 7411, `--triggers result` (a local variant that fires on A1 2.5–3.5 s after the run's result proposal), `--keep`. The run passed 4 of 4 tasks. In two tasks, K1 landed 1.5 s and 2.0 s after `run.result.accepted`. On the next startup, each run recorded "Server restart released the accepted native result of a stopped controller for finalization" (`controller_process_dead`). Each finished `succeeded` about 1 s later, with its task `done` and exactly one run per task (no extra provider turn). Before this fix, the same shape stayed `running` until the 10-minute budget ran out (seeds 7411, 7422, and 7423 from the RC-33b investigation in #19691). ## Risks - Startup could take a lease from a finalizer that is still running. Mitigations: the lease must belong to another boot, the controller process must be proved stopped on this host (the same proof the existing runner takeover uses), and the update matches the exact owner. Remote workspaces are excluded. Multi-host deployments keep the existing PID-liveness assumption that `claimNativeRestartRecoveries` already makes. - Low behavioral risk otherwise: finalization uses the existing reconciler path, which #19591 already exercises after a graceful drain. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, run as an agent in Claude Code with tool use (shell, file editing, local test execution, live restart-chaos waves). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
08f7ac03dc |
fix(recovery): keep a never-started task waiting through a temporary accounting hold (#19702)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Admission checks every wake. A short company accounting hold can
refuse new work after a control-plane restart.
> - When the hold refused a new task's assignment wake, stranded-work
recovery found the task with no run while the hold was still active. It
moved the task to `blocked` with a board recovery comment.
> - A `blocked` task leaves the stranded-assignment sweep, so nothing
woke it after the hold cleared. The task stayed `blocked` with no run,
no interruption and no Retry.
> - The hold was temporary. Tasks created a few seconds later (after the
sweep) started normally once accounting settled.
> - This pull request keeps a never-started task `todo` while a
temporary budget hold lasts. The existing assigned-todo liveness
dispatch then admits it once the hold clears. The task shows the hold it
waits for.
> - The benefit is that a restart no longer silently drops new tasks
created during the accounting settle window.
## Linked Issues or Issue Description
**What happened?**
In a restart-chaos storm on master `06bbcea12f`, three new tasks were
created while the company accounting hold was active. Each assignment
wake got 409 "Company cannot start work while completed runs await
accounting." (or "...native run recovers its unfinished accounting.").
One task rested `blocked` with 0 runs, interruption `null` and
`canRetry=false`. The other two ran later.
**Root cause**
`reconcileStrandedAssignedIssues`
(`server/src/services/recovery/service.ts`, the
`readInvocationBudgetBlock` branch near line 4826) escalates any
assigned task to `blocked` when a budget block exists. It does this
before the assigned-todo no-run branch, which already skips a
budget-blocked task and dispatches it later. So a task that never
started was treated as failed recovery work. The refused wake itself is
stored as a `skipped` `budget.blocked` diagnostic, which is correct; the
escalation removed the only path that re-admits it.
**Expected behavior**
A wake refused by a temporary admission hold waits and is admitted once
the hold clears, with one run. A real hard stop is still a board
decision. Removed assignees and cancelled tasks still do not wake.
**Steps to reproduce**
Create a company with a hard-stop budget and a completed run that has
`cost_accounting_pending = true`. Assign a new `todo` task. The wake
gets 409. Run the stranded-assignment sweep. Before this change the task
becomes `blocked` and never runs.
Related: #19700 bounds the accounting hold itself. This PR is
independent: it handles what happens to a task whose first wake the hold
refused.
## What Changed
- Stranded-work recovery skips a `todo` task with no run while a
temporary budget hold blocks admission. It does not escalate it to
`blocked`. The existing assigned-todo liveness dispatch admits the task
once the hold clears.
- A hard stop (spend at the limit, or blocked unpriced usage) keeps the
existing board escalation. Recovery checks this with the read-only
`policyBlocks` for the policies of the blocking scope. It does not
change budget hold semantics.
- The task interruption projection now covers a `todo` task with no run
whose latest wake was refused with `budget.blocked`. It shows "Waiting
for the budget to allow new work. The task starts automatically when it
does." with no Retry. A budget pause shows the existing "budget" user
action.
- The no-run projection applies the same person-owned gates as the
run-backed projection: a budget pause shows the "budget" action, and an
ordinary agent, project, company or task pause shows "resume". Only an
accounting hold shows the automatic-start text.
- `ExecutionBlockerNotice` shows that wait text for a `working`
presentation that carries a wait reason.
- `doc/execution-semantics.md` records the rule.
## Verification
- New embedded-Postgres regression
`server/src/__tests__/heartbeat-refused-assignment-wake.test.ts`:
- hold active, new task assigned, wake refused (409, `skipped`
`budget.blocked`), two recovery sweeps: the task stays `todo` with no
run and shows the wait reason. Before the fix this failed with `expected
'blocked' to be 'todo'` (the CHA-9 shape).
- hold cleared, two sweeps: exactly one provider turn starts, from
`issue.assigned_todo_liveness_dispatch`, and the task finishes. A later
sweep does not start another turn.
- counter-tests: a terminated assignee and a cancelled task get no run
after the hold clears.
- Existing `creates a board recovery action for budget-blocked assigned
work` (real hard stop) still passes unchanged.
- `pnpm exec vitest run
src/__tests__/heartbeat-refused-assignment-wake.test.ts
src/__tests__/heartbeat-process-recovery.test.ts
src/services/task-interruption-presentation.test.ts
src/__tests__/heartbeat-issue-liveness-escalation.test.ts` (server) and
`src/components/ExecutionBlockerNotice.test.tsx` (ui).
- Server typecheck and `pnpm check:token-gates`.
- Live restart-chaos storm (local, codex/claude/opencode, 6 in flight,
K2 every 3-4 minutes for 20 minutes, no RC-06 mask), revision
|
||
|
DottaandClaude Opus 5.5
|
ece09c156a |
fix(runner): keep an accepted result when ACPX reports its turn lost on restore (#19691)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native Codex, OpenCode, and ACPX runs survive control-plane restarts: runnerd restores the conversation, and the control plane continues the task from its checkpoint > - #19658 made recovery keep a durable result when a Codex restore loses the turn that proposed it. It covered the recovery read path only > - ACPX reports the lost turn differently. After the restored conversation reopens, a live `turn.failed` arrives with `provider_turn_lost_on_restore` > - That live terminal still turned an accepted `done` result into a failed run. The run ended `failed` (`adapter_failed`), and the finished work needed a Retry > - This pull request moves the rule into the shared terminal mapping. A restore-lost turn with a held result for that exact turn settles as completed, whichever path reports it > - The benefit is that ACPX and OpenCode tasks that already finished keep their accepted result after a restart, as Codex tasks now do ## Linked Issues or Issue Description Refs #19658, #19525, #19501, #19539 **What happened?** In a restart chaos wave, an ACPX Claude task (workload A1) wrote its brief and proposed a `done` result (`run.result.proposed`, then a completed `paperclip_finish` tool call). Then a K1 and a K6 crash loop hit the control plane, and `resume_dead_runner` ran at controller generations 2 and 3. The restored ACPX conversation emitted `turn.failed` with `provider_turn_lost_on_restore` (`recoverable: true`) for the checkpointed turn. The control plane then emitted `run.result.accepted` with the `done` result, but `run.terminal` came back `runTerminalState: failed` / `turnTerminalState: failed` / `reportedWorkDisposition: done`. The run ended `failed` / `adapter_failed`, "Native session failed". **Steps to reproduce** 1. Start a native ACPX (or OpenCode) task through runnerd. 2. Let the agent call `paperclip_finish`, and let the control plane persist the proposal. 3. Kill the control plane and runner before the turn ends. 4. Let restart recovery resume the run. The restored runner reports the turn lost on restore as a live terminal. **Expected behavior** Recovery finalizes the accepted result, and the run succeeds. **Actual behavior** ACPX and OpenCode use the same Codex harness session as Codex (`codex-native-backend.ts`). Recovery restored the held result for the turn from the durable proposal. But the live `turn/completed` notification with `status: failed` and `provider_turn_lost_on_restore` went through `mapTerminalTurn`, which emitted `turn.failed`. The backend maps `turn.failed` to `runTerminalState: failed`. #19658 handled only the `thread/read` reconciliation path. ## What Changed - `codex-session-terminal.ts`: a new `settleRestoreLostTurnWithHeldResult` helper. When a terminal is `failed` with `provider_turn_lost_on_restore` and the session holds the semantic result that this exact turn proposed, the turn settles as `completed`. `mapTerminalTurn` applies it first, so live notifications and recovery reads follow one rule. - `codex-session-terminal.ts`: the helper also covers `acpx_active_turn_recovery_closed`. Runnerd's ACPX backend (`acpx_provider_backend.rs`, `restore_session_if_needed`) emits that code instead of restoring the conversation when its own state already recorded the accepted result. It is the same restart loss, with the result already accepted. #19525 already keeps accepted results when close cannot prove suspension, and session close still handles `provider_exit_unconfirmed`. - `codex-harness-session.ts`: `reconcile()` keeps only the missing-turn case from #19658. The restore-lost read case and the prior-terminal check now use the shared helper. - `doc/execution-semantics.md`: the restart paragraph now states that the rule covers Codex, OpenCode, and ACPX, and both report paths. ## Verification - Red before the fix, with the real Codex native backend (`createCodexNativeSessionBackend`) and a fake transport through `executeNativeSession`: `pnpm --filter @paperclipai/paperclip-runner exec vitest run src/native-session-runtime.test.ts -t "journaled completion proposal"`. The new `restore-lost live` scenario failed. Green after the fix: the run completes with the journaled result on `turn-1` and a succeeded terminal, and no `turn/start` is sent. The `restore-lost read` scenario passes on both (#19658 already covered it). - Driver red/green: `src/drivers/codex/codex-app-server-driver.recovery.test.ts -t "live restore-lost"`. With a held result, the live lost terminal was `turn.failed` before the fix and is `turn.completed` after it. With no held result, it stays `turn.failed` with the lost cause, so the #19501 continuation is unchanged. - The driver case was extended to `acpx_active_turn_recovery_closed` (red with a held result before the extension, then green). It was found in a live replay where the run ended `failed`/`adapter_failed` with an accepted `done` result. - Suites: `src/drivers/codex/` and `src/native-session-runtime.test.ts` (580 tests), plus `src/backends/` and `src/live/runnerd-restart-continuation.test.ts`, pass. `typecheck:typescript` passes. - Live chaos replays (ACPX Claude): results and a separate new signature are in a comment below. ## Risks - The rule needs a held result whose turn id equals the lost turn. Recovery restores that result only from the durable `run.result.proposed` for the checkpointed turn, so a result for another turn does not settle it. - A restore-lost turn without a held result keeps its failure and the bounded same-conversation continuation. - Real provider failures and other terminal states are unchanged. - No schema, profile, pack contract, or database changes. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
f09b4704b6 |
fix(recovery): keep the agent's work when Retry continues after exhausted or blocked recovery (#19666)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Task recovery restarts agent work after a control-plane restart or a failed run > - When automatic recovery runs out or is blocked, the user clicks Retry > - Retry then started a fresh isolated attempt cloned from committed HEAD, so the agent lost its uncommitted files, its task branch and the worktree Git link > - A restart must never lose the agent's work > - This pull request makes Retry keep the task's own workspace when that is safe, and tells the agent where the old files are when a fresh attempt is still required > - The benefit is that a Retry continues the real work instead of starting over ## Linked Issues or Issue Description **What happened?** After `native_restart_recovery_blocked` or an exhausted recovery generation, a Retry ran in `isolated-workspace-attempts/<company>/<issue>/<run>/workspace`. That directory was a detached clone of the base ref. It did not contain the uncommitted edits from the issue's git worktree. It was not on the task branch. The agent was not told where the old files were. A restart-chaos wave graded this as an R2 failure (`pre_fault_files_lost`). A second defect: when a restart hit the bootstrap of a fresh attempt, preparing the same run again failed with `isolated_task_directory_binding_mismatch`, so the Retry could not finish. **Expected behavior** Retry continues in the task's own workspace with its edits and `.git` intact when no other writer can hold it. When a fresh attempt is really needed, the old directory stays as it was and the agent gets its path and its changed files. Preparing the same attempt again after a restart is idempotent. **Paperclip version or commit** `master` at `e482f0f632`, restart-chaos waves on 2026-11-06. **Steps to reproduce** 1. Start a native task in a git worktree. Let the agent write an uncommitted file. 2. Kill the control plane so recovery ends in `native_restart_recovery_blocked` or an exhausted generation. 3. Click Retry. 4. The successor runs in a fresh clone without the uncommitted file or the task branch. Root cause: - `server/src/services/fresh-workspace-continuation.ts` `canSelectFreshWorkspace` returns a fresh intent for every task Retry that has a system recovery record. - `server/src/services/explicit-native-continuation.ts` (fresh selection in the source loop) then always persisted `fresh_isolated_workspace`. - `server/src/services/fresh-workspace-seed.ts` `materializeFreshWorkspaceSeed` clones the project source at the base ref with `--no-local`, checks out a detached commit and removes `origin`. Uncommitted work and the branch are not carried over. - RC-35: `server/src/services/heartbeat/workspace-preparation.ts` rejected any reusable existing workspace while `fresh` was true, including the attempt directory that the same run had already created and bound. ## What Changed - New continuation mode `reuse_issue_workspace`. At Retry admission, after all stop proofs pass, `retainedIssueWorkspaceForRetry` checks that the task's bound workspace is the single local directory the failed run used, owned by the task, not a shared workspace, not under remote storage authority, with no other lease, failed cleanup or running workspace operation, a settled recovery generation, no workspace-export failure, and a readable Git state rooted at that directory. Then the Retry continues there with a fresh session. Otherwise the existing fresh attempt is kept. - The reuse decision consumes the generation's one alternative attempt (`freshAttemptRunId`), sets the Retry strategy to `continue`, and records `issue.retry_workspace_reused`. - Environment selection pins the reused workspace binding and fails with `retry_workspace_binding_mismatch` if the binding changed before dispatch. - Fresh fallback: the run's continuation context gets `priorWorkspace` (path, branch, bounded list of uncommitted files). Git status is read with optional locks disabled. The prompt tells the agent to inspect it and carry over the work. The old directory is not changed. - RC-35: preparing the same fresh attempt again after a restart reuses the attempt directory that this run already bound. - Reuse checks run Git through the shared workspace scan scheduler with short deadlines, because admission can hold the issue lock. - `doc/task-recovery.md` and `doc/execution-semantics.md` describe the new contract. ## Verification - `server/src/services/retry-workspace-continuity.test.ts` (embedded Postgres, real git worktree with uncommitted edits): - exhausted-generation Retry and blocked-recovery Retry continue in the same worktree, `.git` file and branch intact. Both failed before the fix (readiness was `fresh_isolated_workspace`). - Blocked recovery without a generation and duplicate holds for one run also reuse the worktree. A running operation orphaned by a restart of the stopped source does not block reuse. - The fresh attempt is kept when the workspace is unsafe: live lease, running workspace operation, corrupt `.git`, missing directory, remote authority, shared workspace, running export generation. `priorWorkspace` is omitted for the unsafe writer and shared cases. - `workspace-preparation.test.ts`: - A `reuse_issue_workspace` Retry prepares in the original worktree with its uncommitted file and branch. - RC-35 red/green: re-preparing the same fresh attempt reuses its bound directory and keeps files written before the restart. Another run still gets `isolated_task_directory_binding_mismatch`. - The fresh fallback sets `priorWorkspace` with the old path and `unpublished.txt`. - `environment-selection.test.ts`: the new mode pins the binding and rejects a changed binding or cwd. - `packages/adapter-utils/src/server-utils.test.ts`: the prompt names the retained directory and its files. - Regression files run locally: `workspace-preparation`, `environment-selection`, `task-retry`, `quarantined-task-retry`, `explicit-native-continuation`, `workspace-recovery`. - Restart-chaos (`tests/restart-chaos`, local, B1+B5, codex+claude, K1/K4/K6, concurrency 2, rc06-masked): - Final head: wave `w-20261107T014633-s7206` on `c6776283f3`: 8/8 PASS, all 8 restart-crossing. 5 tasks needed a Retry (t03-t07: codex B1 K6, codex B1 K1, codex B5 K6, codex B5 K1, claude B5 K6). All five finished in the original git worktree with `.git` and every pre-fault file present (`workspaceMovedTo` empty, `lostFiles` empty); no "Retry keeps a fresh attempt" log line. - Wave `w-20261107T004829-s7205` on `29373a6b6a`: 8/8 PASS, all 8 restart-crossing. 3 tasks needed a Retry (t02, t06, t07). All three finished in the original git worktree with `.git` and every pre-fault file present; no "Retry keeps a fresh attempt" log line. (`cc178492ef` only makes the operation filter stricter for run-less jobs.) - Wave `w-20261107T000412-s7204` on `d197082107`: 8/8 PASS, 7 restart-crossing. 3 tasks needed a Retry (t02, t04, t06). All three finished in the original git worktree (`workspaceMovedTo` is empty), R2 passed with `.git` and every pre-fault file present, and no "Retry keeps a fresh attempt" log line appeared. - Wave `w-20261106T231416-s7203` on the first commit: 6/6 PASS, but its t01 Retry still moved to `isolated-workspace-attempts`. That instance was not kept, so the exact rejection was not recorded. The likely cause is a `running` workspace operation that a killed server left on the stopped source (the source restarted three times). The second commit treats such restart orphans as not live and logs the rejection reason. A `--replay` of that t01 on `d197082107` finished in the original worktree with its `.git` file and no attempt directory. - Review follow-ups in `d197082107`, `29373a6b6a`, `cc178492ef` and `c6776283f3` (run-less workspace jobs still block reuse; red/green covered by the `running_operation` case): holds are grouped by source run; the readiness fallback (blocked recovery without a generation) also reuses the task workspace; Git probes go through the shared scan scheduler with 10-15 s deadlines; `priorWorkspace` is only set for a task-owned directory with no other writer; the reuse rejection reason is logged; `priorWorkspace` uses the same task-owned, settled, single-writer predicate as reuse; Git probes share one budget per workspace; the RC-35 check is a pure path check; renames are parsed from either porcelain column. ## Risks - A Retry now writes into the old directory instead of a new one. The checks above prevent a second writer. A stale binding fails the run with `retry_workspace_binding_mismatch`; it does not write to another directory. - If the cause of the failure was inside the workspace itself, reuse could repeat it. Workspace-export failures, restore failures and required workspace recovery keep the fresh attempt. - A workspace operation of the stopped source that started before the source settled and before the current server process no longer blocks reuse (a restart orphan). Operations with no run, of another run, or started in the current process still block it. - `priorWorkspace` is an optional new field in the continuation envelope. Older adapters ignore it. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
c37bc749ac |
fix(native): keep restart continuations alive through early results and repeat restarts (#19699)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs continue a turn lost to a control-plane restart once, in the same conversation (#15297, #19501) > - In restart storm 3, two Codex runs each crossed one restart and still needed a Retry > - The continuation was attempted, but the driver failed it with `turn_binding_mismatch`, and the run then reported "runnerd exited before provider turn startup" > - A second restart that loses the continuation turn itself also needed a Retry, because the one-shot marker was already used > - A restart should not cost a click when continuing is safe, and a failure should name its real cause > - This pull request fixes the start race, reports the driver's own failure, and gives each admitted restart its own continuation > - The benefit is that interrupted tasks finish without a Retry, within the existing recovery budget ## Linked Issues or Issue Description Refs #19501, #15297, #19658, #19539 **What happened?** Restart storm 3 (seed 7103, RC06 mask on, load 9–12). Codex B5 Retry runs crossed one K1 restart mid-turn. After the restart, recovery resumed the dead runner. The runner restored the conversation and recorded `turn.failed` with `provider_turn_lost_on_restore` (`recoverable: true`). The control plane started the same-conversation continuation (`resumeInterruptedTurn` → `startTurn`). About 1 s later the transport closed with `protocol_failure:turn_binding_mismatch`. The run failed as `native_runner_process_exited: runnerd exited before provider turn startup`. The task needed another Retry. I reproduced it with the focused cell (`--workloads B5 --faults K1 --triggers retry`) and a temporary diagnostic. The failure is "Provider semantic result did not name the active turn": a `paperclip/runResult` reached the Codex driver while the continuation's `turn/start` was still pending and no turn was bound. The driver waits for a pending start before it handles `turn/completed`, but not before it handles `paperclip/runResult`. **Expected behavior** After a restart that loses a turn, the run continues automatically when that is safe. Each admitted restart attempt can continue once, within the provider-attempt budget. A failure names its real cause. **Steps to reproduce** 1. Run `node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env local --adapters codex --workloads B5 --faults K1 --triggers retry --tasks 2 --concurrency 2 --seed 4040 --rc06-masked`. 2. When a lost turn's continuation receives a semantic result before its start settles, the run fails with `turn_binding_mismatch` and the task needs a Retry. The race is intermittent: 1 of about 14 lost turns in my runs. **Root cause** - `packages/paperclip-runner/src/drivers/codex/codex-session-notifications.ts`: the `paperclip/runResult` branch checked the active-turn binding without waiting for `state.turnStartSettled`. The `turn/completed` branch already waits. - `codex-harness-session.ts` `startTurn`: the driver's own protocol failure closed the transport. The pending start then reported the runner exit, which hid the cause. - `codex-session-terminal.ts` `mapTerminalTurn`: a consumed one-shot marker blocked every later continuation in the run, even when the newly lost turn was that continuation itself. The one-continuation allowance is per run, not per issue or session lineage: attaching a new run resets the marker. The storm runs did attempt their continuation; the start race made it fail. ## What Changed - Codex driver: a `paperclip/runResult` waits for a pending turn start to settle before it is bound, like a terminal. - Codex driver: when this session's own protocol failure ends a pending turn start, `startTurn` throws `codex_turn_start_protocol_failure: <code>: <message>` instead of the runner exit. CodexRpc errors and protocol-integrity errors keep their own types. The server now classifies this as `native_session_interrupted` instead of `native_runner_process_exited`; both are retryable. - A later restart that loses the continuation turn itself re-arms the one-shot marker: the driver resets it as it was before the first continuation. It does this when it maps the lost terminal, including during recovery from a checkpoint taken while the continuation ran. The checkpoint gate `nativeRestartInterruptedTurnId` stays strict: a consumed marker still means the allowance is used. Each continuation still needs an admitted dead-runner restart. The existing provider-attempt budget bounds the repeats, and an exhausted budget still blocks with Retry. - `doc/execution-semantics.md`: an additive paragraph. ## Verification Red before the fix, green after: - `codex-app-server-driver.events.test.ts` "binds a semantic result that arrives before its pending turn start settles". Red: `session.failed`. Green: the result binds to `turn-1` and the turn completes. - `codex-app-server-driver.events.test.ts` "names its own protocol failure when that failure ends a pending turn start". Red: "runnerd exited before provider turn startup". Green: `codex_turn_start_protocol_failure: turn_binding_mismatch`. - `codex-app-server-driver.recovery.test.ts` "offers another restart continuation when a later restart loses the continuation turn itself". Red: `null`. Green: `turn-2`. A checkpoint with a consumed marker still returns `null`. - `codex-app-server-driver.recovery.test.ts` "re-arms the continuation when recovery finds the checkpointed continuation turn lost". This recovers from a checkpoint taken during the continuation, then starts the next continuation. Red without the re-arm (the marker stays consumed); green with it. - `native-runner-restart-recovery.integration.test.ts`: the same checkpoint (continuation active, consumed marker naming it) is claimed as `resume_dead_runner`. This is regression coverage. Suites run locally on the final code: - Runner: Codex driver, `native-session-runtime`, `runnerd-restart-continuation` and contracts (745 tests). - Server: `native-runner-restart-recovery.integration.test.ts`. - `tsc` for the runner tests and the server. - A few real-process tests timed out at about 6 s while six live agents ran. Each passed when rerun alone. The settlement suite passed both with and without this change at the same load. Live (local, real Codex, macOS arm64), focused restart-chaos cell `--adapters codex --workloads B5 --faults K1 --triggers retry,time --tasks 6 --seed 4042 --rc06-masked`: 6/6 crossed a restart and passed. R1 passed for all 6, so each continued without a Retry. ## Risks - A run can now continue more than once in the same conversation, once per admitted restart. Each continuation asks the agent to reconcile unfinished actions before repeating them. The provider-attempt budget still caps the repeats. - A semantic result that arrives during a pending start now waits for that start. If the start fails, the result is still rejected by the existing binding check. - The start race was intermittent (1 of about 14 lost turns), so the live cell shows absence of the failure, not a forced repro. The driver regression reproduces it deterministically. - No schema, API or migration change. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use, code execution, and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
9a52a2dfeb |
fix(native): retry ACPX session.open startup timeouts and give cold Claude a 60 s bound (#19661)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runs start an ACPX provider through Runner. `session.open` opens the provider session before any prompt. > - On a loaded host, a cold ACPX Claude `session.open` took longer than its 30-second bound. The controller reported `runner_local_connect_failed: PRP command session.open timed out`. > - The server classified this as the generic `native_session_interrupted` with `ambiguous_state`, so the task was blocked at once. A person had to click Retry, sometimes several times. In one restart-chaos storm, one task needed three Retry clicks, and on another task both Retry runs timed out too. > - The server already has a bounded same-run bootstrap retry for failures that come before provider work (#18854). It did not recognize this timeout. The 30-second bound also cut off normal slow startups. > - This pull request gives cold ACPX Claude a 60-second `session.open` bound, like Pi and OpenCode (#19487). It also retries a startup `session.open` timeout automatically on the same run, from clean state. > - The benefit is that a slow cold start on a busy host completes, or retries itself, instead of waiting for a person. ## Linked Issues or Issue Description Refs #18854 (bounded bootstrap retry for definite `session.open` failures), #19487 (60-second cold OpenCode admission), #19665 (the bootstrap proof that this change reuses). **What happened?** ACPX Claude runs failed with `runner_local_connect_failed: PRP command session.open timed out` about 35 to 40 seconds after they started. The coordinator recorded `native_session_interrupted` with `recoveryMode: "ambiguous_state"` and `providerSessionEstablished: false`. The task was blocked with a board-owned action. No provider event and no checkpoint existed. The host 1-minute load was 17 to 44, sometimes with no restart nearby. **Expected behavior** A normal slow cold start completes. A `session.open` that still times out during runner startup, with no provider event, no checkpoint and no turn, retries on the same run with backoff. At most two automatic attempts follow the first. A run where provider work may have started is never retried automatically. **Steps to reproduce** 1. Run ACPX Claude tasks on a host with a 1-minute load of about 25 or more. 2. Observe a run fail after about 35 to 40 seconds with `PRP command session.open timed out`. 3. Observe the task blocked with `ambiguous_state` and a manual Retry. **Paperclip version or commit** `master` at `6c7cc2c5d6`. ## What Changed The bound (evidence below): - `coldAdmissionTimeoutMs` gives cold ACPX Claude 60 seconds (live adoption keeps 30 seconds). The runner's Rust sidecar `session_open_timeout` and the sidecar's provider handshake (`acpxSessionHandshakeTimeoutMs`) give Claude and Pi 60 seconds too. All three bounds must agree, or the shortest one fails first. Claude's warm run attachment uses the same cold budget, like Pi. The automatic retry: - `nativeSessionFailureSourceCode` returns `provider_session_open_timeout` only for exactly `<startup failure code>: PRP command session.open timed out`. The startup failure codes are `runner_local_connect_failed`, `runner_direct_wss_failed` and `runner_ingress_unavailable`. The transport uses them only before startup completes. After startup the code is `provider_transport_failed`, which is unchanged. - `nativeProviderRecoveryEvidence` treats the cause as pre-session. Durable evidence still decides: a bootstrap retry needs no provider event (`session.started`, `turn.started` and the others) and no checkpoint. The existing budget applies: three attempts with backoff of about 30 seconds, then about 60 seconds. When the budget is used, the task is blocked with the existing "Provider startup failed on 3 attempts before any provider work began" action. - A timeout is indeterminate: the runner can open a provider session after the controller stops waiting. Live runs showed both leftovers, so the retry now starts from clean state: - `prepareNativeSessionOpenTimeoutRetry` (server) runs when the failure settles. It quarantines a local runner root only when the restart-recovery proof (`runnerdStateProvesIncompleteBootstrap`, #19665) shows that no turn began and every recorded process is gone. Any other root, and any remote runner, stays `ambiguous_state` as before. - `resetAbandonedBootstrapRecord` (runner ACPX adapter) runs on a fresh open (no runner-recorded identity). If ACPX has a saved record for the session key with no messages, no prompt and no events, it is marked `reset_on_next_ensure`. Claude saves nothing before the first prompt, so reusing such a record fails the first turn with `session_resume_required`. A record that holds any conversation is unchanged. - A chat retry of an exhausted `provider_session_open_timeout` run is accepted like the other pre-provider causes. - `doc/execution-semantics.md` and the Runner README describe the cause and the bound. ## Verification **Measured startup distribution** (ACPX Claude `runner.session.bootstrap` / `runner.session.startup` spans from restart-chaos run events, 1-minute load 8 to 44): | Source | Bound | Successful cold bootstraps (s) | Failures | | --- | --- | --- | --- | | Earlier storm evidence | 30 s | 2.9, 4.7, 7.4, 8.6, 8.7, 26.2, 27.4, 27.6, 29.2 | startups ended at 34.9 to 40.3 s (the 30 s `session.open` bound plus spawn and `run.prepare`) | | This branch, wave s3601 | 60 s | 6.3, 7.8, 8.1, 13.4, 14.4, 22.3, 22.8, 22.9, **37.3** | 0 `session.open` timeouts | | This branch, wave s3602 | 60 s | 13 samples, p50 10.5, p90 17.9, max 25.7 | 0 `session.open` timeouts | | This branch, wave s3610 (final head, load 28 to 42) | 60 s | 10.4, 11.3, 11.7, 12.0, 13.3, 14.2, 29.6, **30.4**, **30.9** | 0 `session.open` timeouts | Under load, successes clustered just under 30 s and failures sat just above it, so the 30 s bound was cutting the distribution off. With 60 s, cold starts of 30.4, 30.9 and 37.3 s succeeded that the old bound would have failed. 60 s is about 1.6 times the largest observed success, and it matches Pi and OpenCode. **Live restart-chaos waves** (local, ACPX Claude only, 4 in flight, `--rc06-masked`, real provider, K1/K2 restarts): - s3601 (8 tasks, load 19 to 32): 7/8 PASS. No `session.open` timeout. The one failure is an R3 Retry signature after K1 (`native_restart_recovery_blocked`), not this cause. - s3602 (8 tasks, load 17 to 39): 8/8 PASS. No `session.open` timeout. Three Retry clicks were for `native_provider_terminal_failed` (`provider_turn_lost_on_restore`), a different restart signature. - Forced timeout, s3609 (4 tasks, B5, K2): a local, uncommitted fault injection gave every other cold startup a 1.5 s `session.open` deadline. Result 4/4 PASS. Run 4fe6537a timed out, retried itself on the same run, and then succeeded through four K2 restarts with no person involved. Run 02799a85 hit the injection three times in a row; the budget stopped it after attempt 3, and one Retry completed it. - Before the two clean-state fixes, the same forced wave (s3607 and s3608) showed the retry reusing leftover state: `turn.start` failed with `session_resume_required`, and then "ACPX session is not open". Those runs are the red evidence for the two clean-state fixes. - Final head, s3610 (8 tasks, K1/K2, no injection, load 28 to 42): **8/8 PASS**, 0 `session.open` timeouts. The one Retry click (t06) was for `provider_turn_lost_on_restore`, a different restart signature. **Tests:** - Red before the fix: `native-startup-retry.integration.test.ts -t timeout` (embedded PostgreSQL, real executor, real resume claim). The 3 retry cases failed with `native_session_interrupted` / `ambiguous_state`. - Green: `native-startup-retry.integration.test.ts` passes 36/36. New cases: - for each startup failure code, a timeout retries on the same run with backoff, keeps the task `in_progress` with an agent-owned `resume_native_run` wake, and blocks after attempt 3 with a board-owned action; - no automatic retry when `session.started` or `turn.started` is durable, when the code is `provider_transport_failed`, or when another command timed out; - a real runner root with an opened session and no turn is quarantined; a root with a turn command stays `ambiguous_state` and intact; a remote runner is never prepared. - `codex-runtime-adapter.test.ts`: 113 pass. New cases: a 45 s delayed handshake is admitted for Claude and rejected for Codex; a never-prompted record is reset only on a fresh open; records with messages, a prompt or events are kept. - `abandoned-bootstrap-record.test.ts` (real `createAcpRuntime` with `createRuntimeStore` and a controlled ACP peer that, like Claude, cannot load a never-prompted session): without the reset, a fresh open reuses the record and the first turn fails with SESSION_RESUME_REQUIRED after `session/load`. With the reset, the provider receives `session/new` and the turn completes. - `vitest run src/drivers/acpx src/cli` in the runner: 107 files, 1,935 tests pass. - `native-session-executor.test.ts -t "session.open|startup|bootstrap"`: 144 pass. `runnerd-codex-transport.test.ts -t "cold process admission"`: 4 pass. `cargo test -p paperclip-runner-core --lib admission_budget` passes. - `npx tsc --noEmit -p server` and the runner `tsc -p tsconfig.test.json` pass. ## Risks - **This changes Runner Rust code. The Daytona runner image must be rebuilt after merge** for remote sandboxes to get the 60-second sidecar bound. Until then, a remote Claude `session.open` still fails at 30 seconds in the sidecar (as a definite failure, which already retries). A remote timeout stays operator-owned, because a remote root cannot be proven clean here. - A cold ACPX Claude start that hangs now fails after about 60 seconds instead of 30 seconds. - A timed-out `session.open` may have created an empty provider session. No prompt was sent. The retry quarantines that runner root and asks ACPX for a new session. - Same-run retries of a fresh-attempt Retry run can still fail with `isolated_task_directory_binding_mismatch`. This is RC-35 (workspace re-preparation), which has its own fix in progress. - No schema or API change. The new error code appears on runs and in run error codes. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, file edits, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
254b44f72b |
fix(remote-workspaces): recover retired roots that cannot be renamed (#19674)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Remote workspaces must retain saved files across process and sandbox replacement. > - Restore uses identity receipts to distinguish a retired owned root from an unrelated directory. > - Some overlay filesystems reject a directory rename with EXDEV even within one mount. > - That error currently prevents a valid saved workspace from being restored. > - This pull request replaces only a verified retired root in place when that rename fails. > - Runtime state stays in place. Each restore holds a root-specific OS lock. > - A durable workspace epoch fence rejects delayed older commands. A pending marker keeps interrupted cleanup recoverable. ## Linked Issues or Issue Description **What happened?** Workspace restore fails with `remote_workspace_restore_failed: stage=publish code=EXDEV syscall=rename` when the retired directory comes from a lower overlay layer. Retry repeats the same rename. **Expected behavior** Restore should publish the verified saved workspace and preserve runtime state. An interrupted restore should remain safe to retry. **Steps to reproduce** 1. Create an owned retired root with a valid identity receipt, runtime state, and stale files. 2. Make directory rename return EXDEV for the existing lower-layer root. 3. Restore a saved workspace through `restoreRemoteWorkspaceDirectory`. 4. Repeat with an interruption before the new restore receipt is published. **Paperclip version or commit** The regression fails before this change on base `a2aa69bc6230c8b81de3588d7d5571b38db4dd04`. **Deployment mode** Remote sandbox with an overlay filesystem. Local regression: macOS arm64, Node 26.4.0, source checkout. Refs #19231. Refs #19251. Those changes establish the retained-root restore path. This fix handles its EXDEV publication failure. ## What Changed - Keep the EXDEV fallback limited to a verified retired workspace root. Preserve runtime state and the existing ownership checks. - Hold a private SQLite OS lock across receipt checks, file replacement, cleanup, and receipt publication. Keep the lock file outside the replaced root. Process exit releases ownership; a lost reply does not. - Pass the existing workspace authority epoch from the real restore caller. Write and sync a separate epoch fence before root changes. Reject lower epochs, conflicting identities within one epoch, and missing epochs on fenced roots. The fence survives SQLite rollback after process death. - Publish a pending marker without a completed identity until stale preparation-receipt removal and directory-mode cleanup finish. A retry can finish that cleanup. A higher authorized epoch can then restore its selected save. - Preserve later file and permission edits when adopting a completed restore. Refuse unsafe lock files and malformed authority records. Document compatibility and the stop-evidence boundary. ## Verification - RED before the original fix: real restore commands failed with EXDEV. The test preload changes only the kernel rename result. - RED before the review fixes: a live command with an unconfirmed timed-out reply overwrote edits made after retry. Tests also reproduced incomplete cleanup after receipt publication. - RED before the authority fence: a delayed older command replaced a newer completed restore. It also passed after the newer command was killed before root mutation. The final test uses real separate Node processes, OS lock contention, SIGKILL, and a durable on-disk fence. - Final local checks: 142 transport filesystem tests, 16 real PostgreSQL restore-caller cases, and all seven server typecheck layers. The caller cases verify the published receipt uses the current workspace epoch. - Late review RED: an actual separate SQLite owner made the real PostgreSQL restore caller mark an expired lock wait non-retryable. Corrupt fence and receipt bytes also produced a raw syntax error without the required refusal reason. Both regressions now pass. Only the test clock advances the lock deadline; actual OS lock contention and the production caller run unchanged. - Lock contention stays retryable only under the existing same clean writer, epoch, and snapshot proof. Four negative cases change those fields and remain non-retryable. After the owner stops, normal restore returns the useful saved files. Corrupt authority records stay untouched and non-retryable. - Safety cases cover same and different authorized identities, process-death lock release, an unrelated root making progress, pending cleanup interruption, malformed or missing fences, invalid input, and unsafe lock parents/files. - The first final transport attempt passed 139 cases and failed only because the restricted shell could not open a Unix socket (`listen EPERM`). The full suite was then run with the required local socket permission. No production change was made for that setup failure. - Reproduce the filesystem checks from `server/`: `../node_modules/.bin/vitest run --config vitest.config.ts src/services/remote-workspace-transport.test.ts`. Run the caller file with `-t "restores a newer save over this workspace|preserves the published workspace path"` and the repository test PostgreSQL setup. Run `node scripts/typecheck-layers.mjs` from `server/` for all seven layers. - Agent preflight, module boundary audit, and `git diff --check` passed. No added code comments or private issue references occur in the patch. - Local EXDEV proof uses macOS arm64 and Node 26.4.0. It does not claim a new Linux overlay-kernel run or hosted recovery. Final source: [`8092cc6b2ec448ff372337b1e7a43854eae7bb08`](https://github.com/paperclipai/paperclip/commit/8092cc6b2ec448ff372337b1e7a43854eae7bb08). [Exact-head CI](https://github.com/paperclipai/paperclip/actions/runs/42321313715) completed with 59 successful checks, three conditional skips, and one neutral check. Greptile gave this exact head 5/5. The independent review passed. No review threads remain open. ## Risks The EXDEV path changes files in place. It must finish before execution admission. Participating restore commands use a private permanent SQLite lock file and a separately synced epoch fence. The qualified sandbox Node runtime must support `node:sqlite`. Lock contention has a 120-second limit and does not clear live ownership. A legacy command that never acquired this lock remains outside its protection. Hosted recovery must confirm that command stopped; a timeout is insufficient. The new fence does not backfill that evidence. A conflicting identity in the same epoch or a malformed fence stays blocked. A higher epoch can recover an interrupted pending restore only with the existing retired-root replacement authority. No controller database schema, execution permission, or provider-state guard changes. ## Model Used Anthropic Claude Opus 5.5 (`claude-opus-5-5`) authored the recovered patch. OpenAI Codex, GPT-6 family, recovered, reproduced, repaired the review findings, and verified the change with filesystem tests, shell tools, and code execution. The Codex exact serving model ID and context-window size are not exposed in this session. No value is inferred for them. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
6c7cc2c5d6 |
fix(native): do not spend a recovery attempt resuming a cleanly drained run (#19625)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs have an execution recovery budget of three provider attempts, so a run that keeps failing stops retrying > - A graceful shutdown (deploy or SIGTERM) drains native runs: it suspends them for reattachment by the next server > - When the runner did not survive, the next boot resumed the turn and charged one attempt > - Restarts that the run did not cause used up its budget, so recovery stopped and the task needed a manual Retry > - This pull request does not charge an attempt for resuming a run that a graceful drain suspended > - The benefit is that deploys and graceful restarts no longer exhaust a healthy run's recovery budget ## Linked Issues or Issue Description **What happened?** In restart-chaos testing, native runs reached `execution_recovery_budget_exhausted` (3/3) only because of server restarts. In one wave, two restarts crossed each of two runs. In a second schedule (K1, then K2 when ready, then K2 again), each drain correctly suspended the local ACPX Claude run, yet the fourth boot blocked it with `native_restart_recovery_blocked`. Every resume after a drain was classified `resume_dead_runner` and charged an attempt (`nextNativeProviderAttempt` in `native-restart-recovery.ts`, applied by the executor's execution claim). **Expected behavior** A restart that the run did not cause should not consume its recovery budget. A graceful drain records `restart_suspended` and `awaiting_runner_reattach`. The resume that follows should keep the attempt count. **Paperclip version or commit** `master` at `4ef3fa4271`. **Steps to reproduce** Start a native local run. Send the server SIGTERM three times while the run is active, with a full restart each time. The third resume is blocked with `execution_recovery_budget_exhausted`. ## What Changed - `suspendedByCleanDrain()` in `native-restart-recovery.ts` recognizes a run whose most recent recovery claim is older than the drain's `restart_suspended` entry. Ownership waits and deferred blocks between the drain and the next claim do not erase it. The next claim consumes it. - A scan that waits for the old controller to exit (`awaiting_evidence`) records one history entry per reason, so repeated scans cannot push the drain record out of the bounded history. - Restart claims carry `chargesProviderAttempt`. It is `false` for a claim that follows a clean drain. - `nextNativeProviderAttempt()` takes that flag. The executor passes it when it takes the execution lease, so a drained resume keeps the current attempt. - The first execution is always counted: a run drained at attempt 0 (no execution claimed it yet) still charges its first provider start. - A run drained during its third attempt may continue that attempt. The local resume and remote bootstrap gates skip the `>= 3` exhausted check only for a non-charging resume, and the executor still refuses a fourth attempt. - A crash restart (no drain record) still charges one attempt. Reattachment to a live runner still charges none. - `doc/execution-semantics.md` states the rule next to the three-attempt budget. ## Verification - New test in `server/src/__tests__/heartbeat-native-restart-suspension.test.ts` (embedded Postgres, real heartbeat drain and restart recovery): - Red before the fix: the claim after the drain had no `chargesProviderAttempt: false`. - Green after the fix, for drained attempts 0, 1 and 3: the crash claim charges (`true`). After the drain, a controlled backend lets the executor take its execution lease and records the saved `native_run_finalizations.attempt` at provider dispatch: 1, 1 and 3. Attempt 3 is claimed instead of blocked. - Red at the executor: when the executor ignores the flag, the drained attempt-1 resume saves 2. - Ownership-wait case: the drained controller (the test process) is still alive, so two scans wait. The history keeps `restart_suspended` followed by one `awaiting_evidence` entry. After the controller is gone, the claim does not charge, and attempt 3 continues. - Classifier cases (local resume and remote bootstrap): attempt 3 after a crash is still `execution_recovery_budget_exhausted`. Both new cases were red on the previous revision. - `heartbeat-native-restart-suspension.test.ts` 12/12, `native-runner-restart-recovery.integration.test.ts` and `native-runner-ownership.test.ts` 94 passed. - `tsc --noEmit` for `server` passes. ## Risks - A run that itself makes the server shut down gracefully, again and again, would no longer be stopped by this budget. Graceful shutdowns are operator or deploy actions, so this is unlikely. A crash loop (SIGKILL, OOM) still charges an attempt on every restart, and a resumed turn that fails still goes through the existing failure and replacement budget. - This changes only the attempt count. Ownership checks, checkpoint verification and the exhausted gate are unchanged. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test execution). Extended reasoning enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
56c2b28c04 |
Settle saved monitor results after a newer user request (#19644)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native results must finish their durable finalization before later
work can reuse a saved workspace.
> - An accepted result can wait on a monitor while its workspace files
are recovered.
> - A newer user request can replace that old wait before finalization
resumes.
> - The old finalizer then reports lost monitor authority and blocks the
saved-workspace lineage.
> - This pull request proves the newer user request and settles only the
old result bookkeeping.
> - The benefit is released recovery work without replaying an old turn
or changing the current request.
## Linked Issues or Issue Description
Refs #17641 and #16975.
**What happened?**
An older successful native result has saved files and a server-issued
monitor receipt. Its task is now blocked after a newer user-comment
continuation failed during preparation. The old coordinator stays in
retryable_failure with monitor_wait_authority_lost. The retained
allocation ownership check correctly refuses the unresolved predecessor,
so later file recovery also stays held.
**Expected behavior**
The normal finalizer can settle the old accepted result when exact newer
user-request evidence proves the old wait is superseded. It must
preserve the current task, user input, ownership, schedule, accepted
result, and attempt history. It must not start provider work or publish
the old wait summary.
**Steps to reproduce**
Create an accepted successful monitor result and mark workspace
finalization successful. Leave the task blocked. Record a newer
canonical user comment and its ordinary user wake, either queued or
consumed by a same-task continuation that retains the exact comment
revision and source-run lineage. Call the real native finalizer. Before
the repair, both cases remain retryable_failure instead of settling.
**Paperclip version or commit**
|
||
|
DottaandClaude Opus 5.5
|
f4b19e4231 |
fix(runner): finalize a durable proposed result when restore loses its Codex turn (#19658)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native Codex runs survive control-plane restarts: runnerd restores the provider conversation, and the control plane continues the task from its checkpoint > - A hard kill can land after the agent proposed its result (`run.result.proposed`) but before the provider turn ended > - The restored runner then proves that the turn ended with the accepted result. It records `session.reconciled` and `run.terminal`, but no provider turn terminal > - The Codex recovery read did not see that turn as ended, so recovery failed with "persisted active turn … is missing from thread/read". Retries ran out, and the Retry re-ran finished work > - This pull request lets recovery settle that turn with its durable result > - The benefit is that a task that already finished its work continues to finalization after a restart, instead of failing and redoing the work ## Linked Issues or Issue Description Refs #19525, #19591, #19539, #19501 **What happened?** In a restart storm, a native Codex task (B1) proposed a complete `paperclip.run_result.v1` with passing tests. The control plane accepted the completion tool call. About 9 s later, a K1 (SIGKILL and restart of the control plane) killed the controller, runnerd, and the Codex app-server. After startup, `resume_dead_runner` resumed the same run two times. Each resume spent about 25 s in `runner.session.startup` and failed with `native_session_recovery_failed: HarnessReconciliationError: persisted active turn <id> is missing from thread/read`. Then `native_session_retry_exhausted`, replay was blocked (`preserve_without_replay_v1`), and the run ended `native_session_interrupted`. The task needed a Retry, which re-ran the finished work. **Steps to reproduce** 1. Start a native Codex task through runnerd. 2. Let the agent call `paperclip_finish`, and let Paperclip accept the result. 3. Kill the controller, runnerd, and Codex before the provider turn ends. 4. Let restart recovery resume the run. Durable evidence from the kept instance: - The run events have `run.result.proposed` for the turn, then a `completion.tool_accepted` span, then the kill. - The saved checkpoint has `activeTurnId` set to that turn and `semanticResult: null`. - The Codex rollout on disk does contain the turn. The provider did persist it. - The runnerd journal for the first resume has `session.reconciled {previousProviderTurnId: <turn>, activeProviderTurnId: null}` and then `run.terminal {turnTerminalState: completed, runTerminalState: succeeded, reportedWorkDisposition: done}`. It has no `turn.*` event for that turn. The runnerd Codex provider state shows `completedTurnAuthoritative: true` and `completedProviderTurnId: <turn>`. **Expected behavior** Recovery finalizes the accepted result. It does not re-run the task. **Actual behavior** The runnerd Codex transport builds the recovery `thread/read` reply from runnerd's journal. It looked only for `turn.completed|failed|interrupted|cancelled` events that name the turn. Runnerd's "turn ended with an accepted result" branch (`provider_backend.rs`, `recovered_turn_ended_with_result`) emits only the reconciliation and the run terminal. So the reply omitted the turn, and `CodexHarnessSession.reconcile()` threw at `codex-harness-session.ts:796`. The second resume read the same journal and failed the same way. ## What Changed - `runnerd-codex-transport.ts`: a new `runnerdRecoveredTurnTerminal` helper selects the terminal for a restored turn. A provider `turn.*` terminal that names the turn wins, as before, with its cause. A `run.terminal` that follows a `session.reconciled` that names the turn as the ended previous turn (no active turn, no later `turn.started`) now also settles that turn, with the run terminal's `turnTerminalState`. The `thread/read` wait loop uses this helper. - `codex-harness-session.ts`: `reconcile()` now settles a checkpointed active turn that provider history no longer contains when the session holds the semantic result that this exact turn proposed (`resultTurnId` equals the turn). Recovery restores that result from the durable `run.result.proposed` event. Without such a result, a missing turn still fails closed. - `codex-harness-session.ts`: the same rule applies when the restored runner reports that turn `failed` with `provider_turn_lost_on_restore`. This happens when the runner cannot prove the accepted result itself. Later recoveries keep the settled outcome. Without a held result, a restore-lost turn keeps its existing failure and same-conversation continuation. - `fake-codex-app-server`: a test-only `--hold-after-terminal-result` flag keeps the turn open after an accepted completion report. - `doc/execution-semantics.md`: one additive paragraph on this recovery rule. ## Verification - Red before the fix, real `paperclip-runnerd` and fake Codex: `pnpm --filter @paperclipai/paperclip-runner exec vitest run src/live/runnerd-restart-continuation.test.ts -t "durable proposed result"` failed with `native_session_recovery_failed: HarnessReconciliationError: persisted active turn provider-turn-1 is missing from thread/read` at `native-session-runtime.ts:2267`. This is the same stack as the incident. - Green after the fix: the run finalizes on `provider-turn-1` with the proposed result and `runTerminalState: succeeded`. It has no `session.failed` or `turn.failed`, and Codex received only one `turn/start` (no re-run). With the transport fix alone, the test passes in about 5 s. With only the session fix, it also passes (about 30 s, through the 5 s read wait). - Driver red/green: `src/drivers/codex/codex-app-server-driver.recovery.test.ts -t "missing persisted turn"`. The settle case failed before the fix (`recovered: false`) and passes after it. The guard case (the held result belongs to another turn) still fails closed. The existing "fails recovery explicitly when the persisted active turn is missing or replaced" test still passes. - Driver red/green for the restore-lost case: `-t "restore-lost turn"`. The held=true case failed before the fix and passes after it. The held=false case still maps `turn.failed` with the lost cause. - Transport unit test: `runnerd-codex-transport.test.ts -t "reconciled result terminal"` covers the run-terminal rule and its negative cases. - Suites: `src/drivers/codex/`, `src/native-session-runtime.test.ts`, `src/backends/`, `src/live/runnerd-restart-continuation.test.ts`, and `src/live/runnerd-codex-transport.test.ts` pass. Some load-sensitive tests time out at host load 25–30 and pass when rerun alone. The two `proves local suspension after an event backlog … (1024 suffix deltas)` cases also time out on unmodified master. - `pnpm --filter @paperclipai/paperclip-runner typecheck:typescript` passes. - Live chaos replay with real Codex: K1 0–3 s after `run.result.proposed`, 3/3 PASS (R1). Each case resumed the same run, which ended `succeeded`. No Retry was needed and no work was re-run. Details are in the comment below. ## Risks - The transport rule accepts a run terminal only directly after runnerd's own reconciliation that names the ended turn with no active turn. Runnerd emits that pair only after it restored the conversation and proved the turn ended. A provider turn terminal still wins. Other run terminals do not settle a turn. - The session rules use only a result that Paperclip already holds for the exact turn. The control plane accepted that result before the kill. A later provider terminal for that turn cannot change the outcome. The finalizer can already interrupt a provider after an accepted result. - A missing turn without a durable result still fails closed. Its outcome is unknown, and the existing Retry stays available. - No profile, schema, pack contract, or database changes. The Rust change touches only the fake Codex test binary. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
9197abf673 |
fix(native): preserve controller state when remote archiving fails (#19663)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native continuity replacement must preserve the state needed to recover saved work. > - The current replacement moves local controller files before it knows whether the remote archive succeeded. > - A remote failure then leaves the local controller without its recovery files. > - This pull request confirms the remote archive before it moves those files. > - A user turn that never started can use the existing bounded Retry path after verified preservation and cleanup. > - The task retains its saved work and can continue once under current user authority. ## Linked Issues or Issue Description Refs #19638, #19547, #19399 and #19659. The prerequisite #19638 merged as `0819f4ea78dd37fba87341b314be9dac078c1410`. This PR now targets master. A normal dependency integration at `febed950f4` keeps only the six-file archive repair in the PR diff. Fresh master-based CI and review are required before merge. **What happened?** A native continuity replacement moved its local controller journal and provider checkpoint. The following remote archive failed. A later recovery attempt could no longer read the local recovery files. The first-attempt failure also had no normal Retry receipt for the saved user turn. **Expected behavior** A failed or timed-out remote archive retains the local recovery files and starts no replacement provider. A user turn that has not started can request one normal Retry. The existing user, provider-turn, Stop, preservation, cleanup, budget and approval gates remain in force. **Steps to reproduce** 1. Keep a local controller journal and provider checkpoint for a retained remote conversation. 2. Request continuity replacement and fail or time out the remote archive command. 3. Check the local controller files before any later recovery. 4. Settle the native failure under a current user wake and recover the saved workspace. ## What Changed - Confirm remote continuity archiving before moving the local controller state. - Add the exact remote archive failure to the shared bounded user Retry admission from #19638. - Verify failed, timed-out and successful archive ordering with real local files. - Verify ordinary failure settlement and saved-file recovery produce one fresh successor. Keep Stop, system wake and unsettled-provider holds. - Document the file preservation and Retry contract. ## Verification - RED: failed and timed-out remote archives removed the local controller files. Reading those files failed with ENOENT. The saved-user Retry request returned false instead of true. - Initial archive unit verification: eight cases passed, including the three archive outcomes and five existing Cloud archive safety cases. - Initial stacked revision: `9104ee399b`, based on #19638 at `d22d8f9d5ee3dc36550a94d55c1d8376f3b9aab2`. - Final master-based revision: `febed950f4`. All 13 affected cases passed across three Vitest files after dependency integration. They include the seven-mode real PostgreSQL completion case, saved-file successor and negative-hold cases, and real-file archive outcomes. All seven server typecheck layers passed again. Module boundary, whitespace and added-code-comment checks passed. All required CI passed on this exact head. The final snapshot has 59 successful checks, three intended skips, one neutral result, and no pending or failed checks. Current-head Greptile and the independent review are both 5/5, with no open findings. - Final local revision: `9104ee399b`, macOS ARM64, Node 26.4.0, pnpm 9.15.4 and embedded PostgreSQL. The three real-file archive outcome cases passed. The real PostgreSQL completion test passed all seven internal scenarios, including archive failure, prior provider activity, system wake, Stop before and after terminal settlement, and stale runtime promotion. Own Runner/shared/SDK builds and all seven server typecheck layers passed. Nine saved-work continuation and negative-hold cases passed. They verify exported file contents, one user-attributed fresh successor, idempotent hooks, no second automatic chain, and first-use manual Retry for held cases. - `node scripts/check-module-boundaries.mjs`, whitespace checks and the added-code-comment audit passed. - Focused commands from `server/`: Vitest with `-t 'retains the local controller checkpoint until remote continuity archiving succeeds'`; `-t 'replays saved user intent through normal Retry'`; and `-t 'continues a user turn once|does not continue automatically'`. PostgreSQL cases used a fresh local cluster and a 120-second setup hook budget. Application timeouts were unchanged. - The initial stacked head passed all test, build, typecheck and E2E jobs. Greptile and the independent review both scored it 5/5 with no findings. These results do not replace the fresh master-based qualification now running. ## Risks - A remote command can fail after it changed remote state. This change retains local evidence in that case. It does not claim a distributed atomic transaction. - The exact archive error can request one normal Retry only under the existing saved-user admission. The normal path must prove stop, saved files and cleanup before a successor starts. - The protected launch-argument rejection remains enforced. Its exact hosted argument difference was not available through ordinary access. #19659 separately addresses a saved-thread feature-policy argument difference. This PR preserves local archive evidence and does not claim direct hosted verification of that primary cause. ## Model Used OpenAI GPT-6 family through Codex, with reasoning, tool use and code execution. The model is inherited; no override was used. The exact serving model ID and context-window size are not exposed by this runtime. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
0819f4ea78 |
fix(native): verify controller ownership before retrying saved turns (#19638)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native task turns can keep a provider conversation from an earlier run. > - A copied conversation checkpoint names the new run before its controller has attached. > - A failed attachment can make recovery treat that copied identity as current execution authority. > - The current run needs its own durable controller checkpoint, or a valid staged attachment, before exact resume. > - This pull request checks that ownership and uses the normal Retry path to continue a saved user turn after verified cleanup. > - The task can continue once with its saved work and history. ## Linked Issues or Issue Description Refs #19399 and #18860. **What happened?** A new user comment resumed a completed native conversation. The retained runner process was gone before the new controller attached. The new run still had a copied provider session ID. Recovery selected exact resume for that new run and then failed its unchanged owner and lease checks. Ordinary first-attempt failure settlement also omitted the saved-user Retry handoff that restart recovery already uses. **Expected behavior** Exact resume requires current controller evidence. A saved user turn can request one normal Retry after this infrastructure failure. The normal path must verify provider stop, saved files, current user authority and all execution gates before it starts a fresh session. **Steps to reproduce** 1. Finish a native warm run and retain its provider checkpoint. 2. Admit a user comment as a new run and copy the provider checkpoint through the real resume producer. 3. Stop the retained runner before attachment. The real remote process check reports that the process is not running. 4. Observe that the copied provider ID selects exact current-run resume. Observe that ordinary failure settlement saves no Retry request. ## What Changed - Check a bounded encrypted controller header for fenced remote Runner recovery. Keep local Runner recovery on its existing disk-backed path. Match the run, conversation and runner identity. Preserve the existing staged-attachment path. - Share the existing saved-user Retry admission between restart recovery and ordinary terminal native settlement. Keep its exact infrastructure errors, user wake, comment, provider-turn and retry-chain checks. - Add real PostgreSQL and stopped-process regressions. Verify malformed and foreign checkpoint holds, current-owner resume, user authority and Stop. Verify saved-file recovery creates one fresh-session successor. - Document the ownership and Retry contract. ## Verification - RED: the real checkpoint rebind producer and stopped-process check selected exact current-run resume without a current controller checkpoint. - RED: ordinary settlement created zero Retry receipts for the real native failure code; expected one. - Review RED: a real current local controller file existed with no database checkpoint rows. The initial gate returned ambiguous state. The fix takes a required trusted execution-target kind and applies the database requirement only to the remote persistence path. - Initial local revision: `3662055880ccf8e09c3c50a340897e16c78627e7`, macOS ARM64, Node 26.4.0, pnpm 9.15.4, embedded PostgreSQL. - Initial server Vitest: 42 selected cases passed across controller recovery integration, remote restart adoption, ordinary run completion and retained workspace recovery. The cases cover the stopped-process producer, current/foreign/malformed ownership, automatic user continuation, Stop, unsettled turns, system wakes and a manual Retry on first use. - `pnpm --filter @paperclipai/server typecheck`: passed with this worktree's Runner, shared and SDK builds and all seven server layers. - `node scripts/check-module-boundaries.mjs`, whitespace checks and the added-code-comment audit: passed. - Review correction: `8c795e2e0e44b5209da3e228593c5cd35e948cae`. The real local/remote controller case passed against a fresh PostgreSQL cluster. It verifies local disk recovery without database rows, remote current-owner resume, and remote missing/foreign/malformed holds. Fresh `node scripts/typecheck-layers.mjs` passed all seven server layers. The existing offline chat reader now requests metadata only and retains its full independent native-state safety check. - A broader local recheck hit PostgreSQL setup/cleanup hook timeouts. These are retained as failed setup attempts; the fresh-cluster regression above passed. Required CI supplies broad final-head coverage. - Settlement-order correction: `dafb721c3bbb7ae86fbc3c9c6c953ace29ca088b`. The real PostgreSQL regression first proved both failures: Stop after an already-failed row created one Retry instead of zero; a DB-native promotion with an old legacy caller snapshot created zero instead of one. All six settlement scenarios now pass. The hook requires the local failed outcome and the canonical settled native row. Fresh seven-layer server typecheck passed again. - Required shared fixture base integrated at `d22d8f9d5ee3dc36550a94d55c1d8376f3b9aab2`. The six runner-skills fixture cases and all seven server typecheck layers passed. Required CI passed for this head: [workflow](https://github.com/paperclipai/paperclip/actions/runs/42309419592), including all server layers, tests, build, Runner verification and E2E. The final snapshot has 57 successful checks, three intended skips and one neutral result. Current-head Greptile is 5/5; both concrete review threads are resolved. The independent re-review ended without a final response; its prior Stop finding has a real RED/GREEN repair. The focused command was run from `server/`: ```sh ../node_modules/.bin/vitest run --config vitest.config.ts src/services/heartbeat/run-completion.test.ts src/services/native-runtime/native-controller-recovery.integration.test.ts src/services/native-runtime/native-session-executor.test.ts src/services/workspace-recovery.test.ts -t 'replays saved user intent|does not assign current-run resume authority|remote controller restart adoption|continues a user turn once|does not continue automatically' ``` ## Risks - A terminal failure of a fenced Runner adds one exact-run header read. The read selects one encrypted header and limits inspection to 1 MiB. Invalid or missing ownership evidence stays held. - Saved user input can request one automatic Retry. It uses normal preservation, current authorization, budget, approval, Stop and cleanup gates. Unsettled provider turns and system wakes do not start an automatic successor. - This PR changes no provider process, credential, model or lease authority checks. ## Model Used OpenAI GPT-6 family through Codex, with reasoning, tool use and code execution. The model is inherited; no override was used. The exact serving model ID and context-window size are not exposed by this runtime. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
c9391720a1 |
fix(heartbeat): send saved follow-ups after task completion (#19600)
## Thinking Path > - Paperclip manages agents and their work. > - Task messages remain in a durable queue while an earlier run saves its result. > - That result can mark the task done before the user sends the saved messages. > - Send records current user intent, but its wake guard allowed only active task states. > - The old queue therefore remained pending with an obsolete finalization message. > - This change admits that exact Send receipt and reopens the task through the issue service in the successor transaction. > - The agent can receive the saved input once while the completed result remains unchanged. ## Linked Issues or Issue Description **What happened?** Send on a saved message can leave the message queued after its native predecessor succeeds and commits. The old wait message remains visible. Allowing admission alone also fails: continuation construction rejects a task that remains done. **Expected behavior** Explicit Send must deliver the saved input once through normal admission. It must retain the previous run and its result. **Steps to reproduce** 1. Save a follow-up while a native run finishes its result. 2. Let that run succeed, commit its accepted result, and mark the task done. 3. Send the saved follow-up. Before this change, the receipt stays deferred. Related public work: Refs #19542. That change selects recovered workspace files. This change handles the later saved-message admission. Refs #19602 for the independent workflow fixture repair required by CI. ## What Changed - Allow the exact explicit Send wake guard to accept a done task. Keep automatic recovery and cancelled task guards unchanged. - Reopen the task through the company-scoped issue service after admission gates pass. Commit reopening, queue consumption and the successor together. Publish the audit after commit. - Add a native succeeded/committed predecessor regression with unchanged saved work and one successor after two resume calls. - Verify real heartbeat startup under the clicking operator. Keep automatic input, cancelled tasks and paused agents held. - Check workspace preflight for the prospective reopened task. Use the normal company-scoped transition if preflight blocks a done task, so its completion metadata is cleared. - Check a successful native predecessor for its committed accepted result, current ownership and pending cleanup. Keep approvals and questions effective even without a recovery action. - Document the saved Send contract and update the queue test dependency. ## Verification - Final merge qualification for `675914f0fa9b5eb1d2c9c8fcd448e0669d0c4836`: all 56 applicable status checks pass, with no pending or failed checks. [Final CI](https://github.com/paperclipai/paperclip/actions/runs/42307623559) includes typecheck, build, all applicable test and E2E shards, and aggregate verification. Current-head Greptile is 5/5; all five review threads are resolved. Any paused or queued application review is not claimed as a new completed review. - Final base qualification at `675914f0fa9b5eb1d2c9c8fcd448e0669d0c4836`: official preflight passes (1.5s); all six integrated model-normalization cases pass (19.07s); both affected test typecheck layers pass. The integration changes no production server/shared/Runner/adapter inputs, so the preceding 12-case Send proof, two ordinary status guards, and unchanged production typecheck layers remain applicable. Fresh final-head CI supplies broad coverage. - New review regression: both failed and cancelled native predecessors on a task manually marked Done fail before the repair with `continuation_task_ownership_changed` through real `resumeQueuedCommentInterrupt` admission. At `bfb0eff7941090ac3ac43ce4366ba2e3d46612e9`, all 12 affected Send/source/cleanup/workspace/decision cases pass (246.80s), ordinary Done/cancelled rejection cases pass (2; 100.31s), and all seven server typecheck layers pass. History validation uses the prospective reopened status only for an already-verified explicit native Send. No status is changed during a dry run. - Prior `e64e4bb1` CI passed the affected serialized server suites, including the complete saved-message route suite, typecheck and build. General server shard 4 still failed two unrelated stale default-model expectations (`gpt-5.6-sol` versus the new `gpt-6.1-sol` default). The equivalent repair has already merged in #19610 as `8621bcf53ee2e4bbefb7d193add42dd544ca06e8`; the separately prepared #19645 is closed as superseded. This source fix does not weaken or edit those assertions. - Original RED: the committed-source and completed-task startup cases left the saved receipt deferred. - Review RED: Done plus an unrunnable isolated worktree created one successor instead of zero. A successful committed predecessor with pending lease cleanup also created one instead of zero. The new Done-to-blocked path retained stale completion time. Pending approvals and questions each allowed a successor. All failures were captured before their respective repairs. - Integrated production revision `99018264dd1ef002ff3500378184bcd167d9206b`: 417 affected cases pass across the complete native continuation, responsible-user startup, and workspace preflight suites (386 + 18 + 13). These include the original successful startup case and six cleanup/finalization faults that hold the exact saved receipt until settlement, then deliver it once. - `pnpm --filter @paperclipai/server typecheck` passes all seven layers on that integrated head, including the local Runner/SDK dependency build. Module-boundary, whitespace, and added-code-comment checks pass. - Normal master integrations preserve branch history. No force push or assertion weakening was used. Fresh CI and automated review are running on `675914f0fa9b5eb1d2c9c8fcd448e0669d0c4836`, which normally integrates the already-merged #19610 fixture repair after the reproduced inherited CI failure. - The earlier CI attempt failed in five unchanged workflow fixture cases because IMAGE_ONLY was unset. The merged #19602 repairs that fixture. A later inherited duplicate default in that independent fixture has a separate repair; this PR does not edit it. - Follow-up route-fixture RED: three existing native queue delivery cases failed because their fixture marked a run successful without an accepted result or committed finalization. The same three cases pass after the fixture records the complete native success contract (42.08s); delivery assertions and production gates are unchanged. The affected test-tree TypeScript layer also passes. The full route run passed all 119 assertions but failed its 30-second database teardown hook, so it is not recorded as a suite pass. That result is retained. One unchanged bounded rerun was stopped after more than 15 minutes without a final verdict; it is inconclusive, not a pass. - Final integrated head `e64e4bb1`: all 13 focused route and completed-task admission cases pass across two real-PostgreSQL suites (188.95s), including the three repaired route fixtures and every new cleanup/finalization/decision hold. Fresh preflight passes; all seven server typecheck layers pass. Required final CI supplies the broader route and server coverage. - These tests use real PostgreSQL and normal admission, cleanup, queue consumption, and startup services. The startup suite controls the provider boundary. Hosted provider acceptance remains pending normal deployment. ## Risks - Explicit Send can reopen a done task. This is limited to its validated saved receipt and current assignee. A cancelled task remains held. - Reopening uses the existing company-scoped issue transition and shares the successor transaction. A failed admission does not reopen the task. Provider Stop, workspace, approval, pause and budget checks retain their existing owners. - Cleanup and pending decisions retain the done task and saved receipt. An unrunnable workspace uses the existing blocked-task diagnostic. No schema or runtime package changes. ## Model Used - OpenAI Codex, GPT-6 family; exact serving snapshot/context window not exposed; repository analysis, code editing, tool execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
795c3c4c1d |
fix(recovery): continue saved input after historical retry pause (#19576)
## Thinking Path
> - Paperclip manages work performed by AI agents.
> - Its scheduler parks work when another run holds a shared workspace.
> - A pause can refuse the scheduled retry before provider dispatch.
> - Older schedulers recorded that refusal in the run log without a JSON
marker.
> - After the pause was released, a saved user message could remain held
for execution recovery.
> - This change verifies the original refusal and parent workspace-wait
proof before admitting one continuation.
> - Existing execution, Stop, budget, approval, and workspace guards
remain in force.
## Linked Issues or Issue Description
Refs #19466, #19552, #19580, #19602.
**What happened?**
A saved user message remained deferred after a pause was released. Its
source was a never-claimed workspace retry. The historical source had an
`issue_paused` error and no JSON cancellation marker. The continuation
proof accepted only a different system-cancellation shape.
**Expected behavior**
The normal saved-message worker admits the existing input once when
trusted scheduler evidence proves no provider dispatch occurred and
current admission gates permit work.
**Steps to reproduce**
1. Record a host-certified pre-provider workspace wait and its scheduled
retry.
2. Let the real scheduler refuse promotion under an active task pause,
including after the user presses Retry Now.
3. Keep the historical run-log-only cancellation shape, then release the
pause.
4. Run concurrent normal saved-message recovery scans.
**Paperclip version or commit**
The regression fails on master `f0006c7783`. It leaves zero queued
successors. The branch now normally integrates the fixture repairs in
#19602 and #19628.
## What Changed
- Accept the historical cancellation shape only when a trusted system
lifecycle warning records the exact scheduler pause refusal. A prior
Retry Now event does not invalidate that proof.
- Bind the event to a released same-company pause and its task
membership.
- Reuse the existing parent workspace-wait certificate,
no-provider-output checks, and retained-execution rejection.
- Preserve historical cancellation records and consume the existing
saved input through normal admission.
- Document the contract and add real PostgreSQL coverage using actual
scheduler suppression and pause release.
## Verification
- RED: the real scheduler suppressed the retry under a pause. After
release, two normal recovery scans left zero successors.
- GREEN: the same path creates one successor and coalesces the original
saved request.
- All 389 tests in the complete native continuation suite pass before
the final base update. The only changed continuation source from master
is the separately merged one-line repair in #19552.
- Fourteen focused cases passed at
`2aaeb50f78ba0a22809ab96a00076ce50fcea559`, including a parent pause
covering a child task. Thirteen cases also passed before the base
update. They cover the normal path and missing event, wrong task,
missing membership, active or mistimed pause, provider activity,
retained lease, operator Stop, and invalid ancestry.
- The earlier seven-layer server typecheck passed after building this
worktree's Runner and SDK dependencies. The current review fix passes
the exact CI test-layer typecheck. Fresh required CI will supply
current-head source coverage.
- The first sandboxed database run could not bind loopback. A later
fresh dependency install omitted native PostgreSQL library symlinks and
failed before executing tests. The package's own postinstall hydration
repaired that setup. Previous-head verification passed all 14 focused
cases at `2aaeb50f78ba0a22809ab96a00076ce50fcea559`.
- Module-boundary and whitespace checks pass.
- Commands: `cd server && pnpm exec vitest run
src/services/explicit-native-continuation.test.ts`; `pnpm --filter
@paperclipai/server typecheck`.
- Environment: macOS arm64, Node 26.4.0, pnpm 9.15.4, real local
PostgreSQL.
- Previous head `2aaeb50f78ba0a22809ab96a00076ce50fcea559` had Greptile
5/5 and 56 passing checks, with three skipped. A later advisory review
found the Retry Now event-order case. Those checks do not qualify the
new head. Hosted recovery remains unverified.
- Review RED: the real `retryScheduledRetryNow` producer wrote
lifecycle/info at sequence 1 and the exact pause warning at sequence 2.
After real hold release and two concurrent normal queued-message scans,
both task-rooted and parent-rooted cases produced zero successors
instead of one. Both failed before the production change (17.24s).
- Review GREEN: all 17 focused cases pass at
`f38e9934b8ce3f7ddfb23eaa0f979c3f44e7a990` (4.74s). The previous 14
cases remain covered. Two new cases use the real Retry Now producer, and
one new case rejects contradictory provider execution evidence. No
pause, membership, timing, ancestry, output, retained-execution, or Stop
check was removed.
- Current focused command: `pnpm exec vitest run
server/src/services/explicit-native-continuation.test.ts -t 'historical
pause|Retry Now pause history' --fsModuleCache`. Current test-layer
typecheck: `cd server && node ../node_modules/typescript/bin/tsc
--noEmit --project
node_modules/.cache/paperclip-typecheck/tsconfig.tests.json --pretty
false`. Both pass.
- Final-head verification: all 17 focused cases passed again at
`c88687d69f2c87fe1b81b5fffd670e3cf26a884d` (8.00s). The prior exact CI
test-layer typecheck qualifies the unchanged typed logic; fresh CI
supplies broader final-head coverage.
- The follow-up commit removes only the new code comments to follow
CONTRIBUTING.md. Recovery and test behavior are unchanged. That revision
had Greptile 5/5 with its actionable review thread resolved. It is
retained as historical verification; current-head qualification is
below.
- The first final-head CI attempt failed in the unchanged CLI company
import/export fixture before assertions: its child server did not reach
the runtime-info health check within 30 seconds. Other suites passed.
The exact fixture passes locally with its original timeout. The single
failed-shard retry passed at the unchanged head, together with its
dependent CI checks. [Original failed
job](https://github.com/paperclipai/paperclip/actions/runs/42279773036/job/128106195975).
No assertion or timeout was changed.
- The independent required Runner support failure reproduced five
`IMAGE_ONLY: unbound variable` errors. #19602 repaired its fixture
inputs and merged as `4d554f08218b04df49ec0f2a0d5c5dfdfaf65660`. This
branch normally merges that master with no conflicts or force push. The
current host-preparation log verifier from #19580 remains intact.
- Final integrated head `ffc9e8d9516343a9823b4b80b5fcfccc548844ba`:
17/17 focused recovery cases pass (12.71s), all nine workflow-target
cases pass through the official Runner Vitest config (1.10s), and all
seven current server typecheck layers pass. Worktree preflight rebuilt
current Runner dependencies and reused the Cargo seed; it completed in
61.3s with Rust debug binaries skipped.
- Prior integrated head `ffc9e8d9516343a9823b4b80b5fcfccc548844ba`
finished with 54 successful checks. Its only code-check failure was a
global unhandled readiness rejection after 2,285 assertions passed in
the server8 shard, plus dependent verify. The real-child race is fixed
separately in #19624; it was not hidden by a rerun here.
- Final normal integration head
`550d1732b6960d6b349d03903f448fa1d71c17f6`: exact 17 paused-retry
lifecycle regressions pass (71.06s), all nine official workflow target
cases pass (2.43s), all seven server typecheck layers pass, and
module/whitespace checks pass. Required preflight completed in 45.641s.
Current preparation-log guards from #19580 remain intact. The diff
against master contains only the original three owner/test/doc files.
- Prior head 550d had two inherited model-default assertion failures
plus a retained-cleanup backlog 15s timeout. The exact unchanged cleanup
case passes locally (41.91s total including DB/hooks); no timer or
assertion was weakened and no old CI job was rerun. The model fixture
repair was independently merged in #19610 at
|
||
|
DottaandPaperclip
|
c09fb60ca6 |
fix(recovery): retain worker-stop audit in preparation retries (#19612)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Recovery must preserve saved files before it admits another execution. > - A failed preparation can need a verified stop of its previous workspace worker. > - Recovery keeps that verified stop receipt in the run profile for audit. > - The preparation retry guard treats the retained receipt as unknown execution evidence. > - This pull request recognizes a valid same-run audit receipt while it retains all current execution and file checks. > - A saved Retry can then continue once after recovery completes. ## Linked Issues or Issue Description **What happened?** A native preparation failed before provider dispatch. The recovery worker verified that the previous workspace worker had stopped and saved its files. The same HTTP Retry stayed deferred because the run profile retained `nativeWorkspaceOwnerStopProof`. **Expected behavior** The saved Retry should continue once after all preparation, stop, file, ownership, and authorization checks pass. The old run and its audit receipt should remain unchanged. **Steps to reproduce** 1. Fail native preparation with a retained remote workspace and no provider activity. 2. Let the recovery worker acquire the old workspace owner through verified platform stop evidence. 3. Submit one HTTP Retry and let preservation complete. 4. Observe that the old guard retains the retry hold even after saved-file verification passes. **Paperclip version or commit** Reproduced against `d728be8fbf`. Related work: #18908, #19021, #19026, and #19466. This change covers the retained workspace-worker stop audit left by a later repair. ## What Changed - Add a typed reader for a host-persisted same-run worker-stop audit receipt. Keep signature and expiry validation at ownership acquisition. - Recognize only that validated audit field in failed-preparation settlement. Keep unknown profiles and active owners blocked. - Exercise real preservation, HTTP Retry delivery, native dispatch, saved files, accepted results, and task completion. Add malformed, foreign-run, unstopped-worker, and active-owner cases. - Document the distinction between retained audit evidence and current execution authority. ## Verification - Failing before the fix: the current-worker preservation case passed; the stopped-worker case produced zero successor runs after the same Retry, where one was expected. - Passing after the fix: 47 recovery cases, 10 native dispatch/preservation cases, and 6 platform-proof tests. The two preservation success cases overlap between the first two groups. - Server typecheck passed all seven layers after the final test changes. The Runner and SDK dependencies were built in this isolated worktree. - Final comment-free commit `44f8f33f8d917dddb51852283a479786102eab4d`: a fresh combined run passed all 61 affected cases in 67.92s. Fresh server typecheck passed all seven layers. - The earlier `95d0376c` combined run passed 60 cases and hit one `socket hang up` in the existing `catalog_provider_warning` HTTP test. That case passed in a bounded isolated run (12.48s), and the final 61-case combined run passed. The socket failure remains recorded as an intermittent HTTP harness limitation. No retry or weaker assertion was added. - Final commit `44f8f33f8d917dddb51852283a479786102eab4d` completed all required CI gates: 57 checks succeeded, 3 intentional skips, and the advisory Paperclip review is neutral with a substantive 5/5 verdict. Greptile is 5/5 and there are no unresolved review threads. [Final CI run](https://github.com/paperclipai/paperclip/actions/runs/42293625696). The follow-up commit only removes six new code-comment lines to follow CONTRIBUTING. - The native provider and signed platform API are controlled test boundaries. The tests use real PostgreSQL, recovery, ownership, file persistence, retry admission, dispatch, and finalization services. Deployed recovery remains unverified until normal rollout. ## Risks - The main risk is mistaking an audit receipt for current stop authority. The reader checks its exact shape and run binding. The receipt cannot replace stopped leases, saved-file receipts, settled preparation, current authorization, or the absence of an active owner or provider evidence. - The negative matrix runs with the audit field present. Provider activity, unknown usage, bad file receipts, Stop, actor changes, and unsettled leases still block continuation. - No schema or migration change is required. ## Model Used OpenAI GPT-6 in Codex. The exact deployment model ID and context-window size are not exposed in this session. Used reasoning, code editing, shell execution, and test tools. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
876a2e4857 |
fix(recovery): recover a prior boot's unstarted run at lease expiry (#19578)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - After a server restart, a task must continue automatically and promptly when that is safe > - A user clicks Retry, and the server is killed before the new run starts a process > - Recovery leaves that run `running` until the orphan reap's five-minute stale-age threshold passes > - The user sees nothing happen for about five minutes > - This pull request recovers that run when its controller lease from the earlier boot expires > - The benefit is that the automatic retry starts about one lease period after the kill, not five minutes later ## Linked Issues or Issue Description **What happened?** A Retry admitted a legacy run. The server was killed before the run recorded a process. The run kept a controller lease from the killed boot and `executionStage: preparing`. After the restart, live recovery skipped the run until `updatedAt` was five minutes old (`server/src/services/heartbeat/recovery.ts`, `staleThresholdMs ?? 5 * 60_000`). Restart-chaos measured 5 m 01 s from run creation to `process_lost` and the automatic retry, twice (4 m 20 s after the server was ready). **Expected behavior** Recovery should claim the run as soon as the earlier boot's lease expires. The lease is the only liveness proof for a legacy controller, and no process was ever recorded. **Paperclip version or commit** `master` at `71313207eb`. **Steps to reproduce** Click Retry on an interrupted task, then SIGKILL the server within about one second. Observe the retried run stay `running` for about five minutes. ## What Changed - `recoverActiveExecutions` skips the stale-age gate for a legacy `running` run when all of these are true: its controller lease is expired (checked by the existing live-lease gate), the lease belongs to an earlier boot (`controllerBootId !== legacyControllerBootId`), the run is still `preparing` (no adapter dispatch), and the run has no recorded `processPid` or `processGroupId`. - The stage and process fields committed with the revocation are checked again. If the old controller dispatched before its lease expired, the run keeps the age guard. - The existing revocation (`revokeExpiredLegacyControllerWithState`) still requires the expired boot ID, so a concurrent renewal wins. The existing `restartInterruptedBeforeDispatch` rule still decides the single process-loss retry. - `doc/execution-semantics.md` describes the rule in section 10. - Two regressions in `server/src/__tests__/live-recovery-work.test.ts` (embedded Postgres, real residual-recovery workers). ## Verification - Red before the fix: `recovers a fresh run from a prior boot at lease expiry ...` failed with `expected 'running' to be 'failed'`. - Green after the fix: `pnpm exec vitest run src/__tests__/live-recovery-work.test.ts -t "stale-age|fresh foreign controller|suppressed live runs"` passes 4/4. The run fails `process_lost` with `executionRecovery.providerWorkStarted: false` within seconds of lease expiry, and exactly one retry runs to `succeeded`. - Race case: a DB trigger makes the row committed with the revocation say `dispatching` while the scan saw `preparing`. The run stays `running`, and no retry is created. - Guard cases: a run with an expired lease of the current boot, a run from an earlier boot with a recorded process, and a run from an earlier boot that was `dispatching` (for example a remote HTTP adapter) all stay `running` after four minutes. - Related suites: `heartbeat-process-recovery`, `heartbeat-observed-workspace-preservation`, `native-startup-cleanup-recovery`, `services/heartbeat/recovery.test.ts`, full `live-recovery-work`. - Live K8 probe (restart-chaos supervisor, local instance, ACPX Claude agent): the first run fails with a provider error, the probe clicks Retry, and the server is SIGKILLed 100 ms later, before the Retry run records a process. The probe then measures how long the Retry run stays `running`. | Build | Retry run created to `process_lost` | Server ready to `process_lost` | Automatic retry run | | --- | --- | --- | --- | | Base (`71313207eb`) | 300.1 s | 293.5 s | started at 5 m 00 s | | This PR | 60.1 s | 54.5 s | started 0.1 s after `process_lost` | The claim now happens when the killed boot's 60-second controller lease expires. In this probe the restart took about 6 s, so the claim came about 54 s after ready. In the restart-chaos waves, kill to ready took about 41 s, so the claim would come about 20 s after ready. - A chaos wave with the `retry` trigger (`w-20261106T195715-s2727`, branch #19513 merged locally) missed the trigger: no Retry was offered because recovery completed the task automatically (claude t01 PASS). The probe above makes the Retry-then-kill timing deterministic. ## Risks - Low risk. The change only removes an age wait. Ownership is still proved by the expired lease of another boot. A rolling deploy or hot restart whose earlier instance is still alive keeps renewing its lease, so the run is not touched. - Runs with a recorded process, and dispatched runs, keep the five-minute guard. A process on another host or a remote request is not inferred dead. - Native runs are not changed. Their ownership is in `native_run_finalizations` and startup native recovery. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test execution). Extended reasoning enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
da9a4cb96c |
fix(native): retire a finished local Runner that stopped before suspending (#19597)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The native Runner path runs each per-turn agent turn through a
runnerd process, and the next turn must prove that the previous Runner
is finished before it takes over the saved session root
> - Under load, a local turn can have its result accepted while its
`runner.suspend` command does not finish, so the Runner's own receipt
stays unsuspended
> - The retirement proof that handles the same case for sandbox Runners
refused every local Runner with `target_not_sandbox_runner`
> - So the next turn failed with `runner_state_identity_mismatch:
prior_owner_terminal_state_indeterminate` and its session root was
quarantined
> - This pull request extends the retirement to a local Runner when the
prior Runner process is proven stopped by exact PID and start time
> - The benefit is that a follow-up turn (for example, after a user
answers a question) continues instead of failing
## Linked Issues or Issue Description
Refs #19525 (keeps the accepted result in the same incident; this is the
next run).
**What happened?**
In the runner E2E cell
`local-session-integrity.runner-acpx-claude.local.structured-question-restart-resume`,
the first turn asked a question and its result was accepted. The Runner
then logged `runner did not prove durable suspension before checkpoint`
(`runner.suspend` pending, Runner lifecycle `ready`). After a server
restart and the answer, the next run failed with
`runner_state_identity_mismatch:
prior_owner_terminal_state_indeterminate;
retirement_refused=target_not_sandbox_runner`, and its root was
quarantined.
**Expected behavior**
When the prior local Runner is provably stopped and every other
retirement proof holds, the next run retires that owner as a continuity
break and continues. If the stop cannot be proven, the run keeps the
named rejection.
**Steps to reproduce**
1. Run a local per-turn native turn whose `runner.suspend` does not
finish after its result is accepted (host load average ~40).
2. Start the next turn for the same session.
3. See `retirement_refused=target_not_sandbox_runner`.
## What Changed
- `server/src/services/native-runtime/native-restart-recovery.ts`: add
`proveNativeLocalRunnerStopped`. It uses the existing restart-recovery
checks: the recorded Runner PID is dead or now belongs to a process with
a different start time; its recorded process group is not alive; each
recorded provider process is dead or recycled. A missing PID, start time
or process group, a live recorded process, or a start time that cannot
be compared is not proof. When settlement already cleared the run's
process fields, the saved `native.local_process_stopped` receipt is the
Runner proof. Extract `nativeProviderProcessIdentities` (checkpoint plus
run events) so restart recovery and this proof read provider PIDs the
same way.
- `server/src/services/native-runtime/native-session-executor.ts`
(`retireFinishedPerTurnRunnerdOwner`, `readSettledRunnerdProjection`): a
local target is now eligible. For a local Runner, the projection accepts
the Runner's own unsuspended receipt for the same identity and
unfinished Runner lifecycle commands (`runner.suspend`, `runner.drain`,
`runner.shutdown`) only. All other proofs are unchanged: committed
result, released lease, no recovery state, stopped controller, delivered
terminal, other commands complete, tool inputs answered, quarantine
copies. The process proof runs before retirement and again just before
the archive.
- `doc/execution-semantics.md`: describe the local case.
- Tests: seven integration cases in
`native-recovered-owner-followup.test.ts` and ten unit cases in
`native-restart-recovery.test.ts`.
## Verification
- Red before the fix: the four new integration cases failed with
`retirement_refused=target_not_sandbox_runner`, which is the incident
error.
- Green after the fix: `pnpm exec vitest run
src/services/native-runtime/native-recovered-owner-followup.test.ts`
35/35. The new cases prove that a stopped Runner is retired (root
archived under `continuity-breaks`, no quarantine), and that a live
recorded Runner (`runner_process_alive`), a missing identity
(`runner_process_identity_missing`), and a pending non-lifecycle command
(`journal_commands_unsettled(turn.start.pending)`) keep the rejection.
All existing sandbox cases still pass, including "keeps an unsuspended
Runner receipt decisive" for sandbox targets.
- `native-restart-recovery.test.ts` unit cases: recycled PID, live
process, unknown start time, live group, live provider, missing PID or
start time.
- `native-session-executor.test.ts` + `native-restart-recovery.test.ts`:
1190 pass. `native-runner-restart-recovery.integration.test.ts` +
`native-restart-recovery-isolation.test.ts` (shared provider-PID
extraction): 81 pass. `tsc --noEmit -p server/tsconfig.json`: pass.
- Live: `pnpm test:e2e:runner -- --max-parallel 1
--max-automatic-retries 0 --id
local-session-integrity.runner-acpx-claude.local.structured-question-restart-resume`,
3 runs at
|
||
|
DottaandClaude Opus 5.5
|
586caa5f49 |
fix(native): hand a shutdown-interrupted startup to the next controller (#19596)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The native Runner path runs an agent turn through a runnerd process that the control plane owns > - A restart of the control plane must keep the task going without a click, if that is safe > - A graceful shutdown closes the HTTP listener first and then drains the heartbeat runs > - A native run that was between provider dispatch and runnerd backend setup in that gap failed with `local_runtime_api_listener_unavailable`, so the drain did not see it as running and did not suspend it for the next boot > - This pull request makes that failure a restart detach: the run stays running with its claim, the drain suspends it, and the next controller adopts it > - The benefit is that a SIGTERM in this window no longer spends a failure and a same-run retry, and the task continues by itself ## Linked Issues or Issue Description Refs #19530 (the same restart window, before provider dispatch). **What happened?** In a restart chaos replay (ACPX Claude, local runner), a restart-recovered run got SIGTERM while `createRunnerdBackendWithinSessionClaim` built the runnerd backend. `resolveLocalRuntimeApiUrl()` threw `local_runtime_api_listener_unavailable` because shutdown had already closed the listener. The run settled `failed`. The drain then logged `restartSuspendedRunIds: []`. The next boot used the one bootstrap retry. A second SIGTERM hit the same window, and the task stopped at a manual Retry. **Expected behavior** A run that this process cannot continue because it is shutting down is handed to the next controller. It is not settled as failed. **Steps to reproduce** 1. Start a native Runner task and restart the server (K1) so that startup recovery resumes the run. 2. Send SIGTERM while the resumed run sets up its runnerd backend. 3. See the run fail with `local_runtime_api_listener_unavailable` and the drain report no suspended runs. ## What Changed - `server/src/runtime-api.ts`: `resolveLocalRuntimeApiUrl()` throws the new `LocalRuntimeApiListenerClosedError` when this process had a listener and closed it (the `listening` flag is false or the `close` event fired). The error message stays `local_runtime_api_listener_unavailable`, so the existing classifiers do not change. A listener that never started still throws the plain error. - `server/src/services/native-runtime/native-session-executor.ts`: for a local Runner, the startup failure handler treats `LocalRuntimeApiListenerClosedError` like a restart detach. It stops lease renewal, keeps the claim and throws `NativeControllerDetachedForRestartError`. Heartbeat already keeps such a run `running`. A remote Runner keeps the same-run bootstrap retry from #19462, because startup recovery has no process evidence to adopt a claimed remote startup (`native-runner-restart-preparation.integration.test.ts` still covers that path). - `doc/execution-semantics.md`: describe the new rule. - Tests: a heartbeat regression that closes the real runtime API listener before backend setup, then drains and restarts; a unit test for the closed versus never-started listener; two `vi.mock` factories keep the real module exports. ## Verification - Red before the fix: `pnpm exec vitest run src/__tests__/heartbeat-native-restart-suspension.test.ts -t listener` failed with `restartSuspendedRunIds: []` and the run `failed` (`local_runtime_api_listener_unavailable`, `bootstrap_retry`). - Green after the fix: the same test passes. The drain suspends the run, the run stays `running` with `awaiting_runner_reattach`, and the next controller claims it and reaches the backend once. - `pnpm exec vitest run src/__tests__/heartbeat-native-restart-suspension.test.ts src/http/server.test.ts src/__tests__/server-startup-feedback-export.test.ts src/services/native-runtime/native-session-executor.test.ts src/services/native-runtime/native-recovered-owner-followup.test.ts`: 1333 tests pass. One timing test in `native-session-executor.test.ts` failed once under host load average ~30 and passed alone. - `pnpm exec tsc --noEmit -p server/tsconfig.json`: pass. - Live, restart chaos harness, local ACPX Claude (`claude-sonnet-5`), `--rc06-masked`, this branch at |
||
|
DottaandPaperclip
|
b55d8a9201 |
fix(runner): retain admitted OpenCode pack on ordinary continuation (#19558)
## Thinking Path > - Paperclip manages AI agents and preserves their work across runs. > - Native OpenCode turns can restore a saved provider session after the prior daemon stops. > - The Rust provider binds that saved state to exact qualified launch paths and artifact bytes. > - Ordinary continuation selected the current provider pack, although live reattachment retained the admitted pack. > - A new proxy or a diagnostics setting could therefore replace the launch profile behind saved state. > - This pull request retains the recorded pack for ordinary OpenCode continuation with a verified durable binding. > - The benefit is compatible continuation without weakening artifact integrity or granting replay authority. ## Linked Issues or Issue Description Refs: #17512, #18237. These added saved authority and compatible pack reuse. This change covers ordinary OpenCode continuation. **What happened?** An ordinary remote OpenCode continuation could select a newer pack while restoring its existing provider session. The Rust recovery guard then rejected the changed launch profile. The same error can occur after a controller release changes the bundled proxy. **Expected behavior** A continuation with a valid saved binding and pack receipt uses the same admitted artifacts. If those artifacts are unavailable, it holds the saved state instead of selecting another pack or creating a provider session. **Steps to reproduce** 1. Admit a remote OpenCode session with a qualified protected provider pack. 2. Persist its pack receipt and suspended durable Runner state. 3. Change the controller bundle to a valid pack with a different proxy digest. 4. Start an ordinary continuation with diagnostics enabled. The new executor regression fails before the repair because selection rejects the available original pack in favor of the new bundle. It passes after the repair. ## What Changed - Retain the saved provider pack for ordinary OpenCode continuation with a verified durable session binding. - Keep provider-pack retention independent of surviving-daemon identity: a new daemon records its current binary, and an authorized continuity break restores normal compatible pack selection. - Test retained pack selection after a proxy upgrade and fail closed when only a newer pack is available. - Test the real Rust executor with changed, valid proxy file bytes. It rejects drift without changing saved state and accepts the original artifacts again. - Document the selection rule and its limits. ## Verification - Initial prevention focused Vitest: 220 passed across native session execution, pack authority, pack verification and pack preparation. The executor regression was observed failing before the production change. - Rust native provider tests: all five OpenCode tests passed. These include the existing drift guard, actual changed proxy bytes, original artifact restoration and restart-lost turn behavior. - Advisory review follow-up: reproduced stale daemon identity in the actual executor and incompatible pack selection after a governed continuity archive. Both fail before their corresponding repairs and pass after them. Added an executor regression for a missing authority receipt; it remains at the current release integrity gate without creating a new receipt. - Updated head `639c080856` integrates master `17901d74ba` by normal merge. Affected Vitest: 202 passed across four files, including all new review regressions; all seven server typecheck layers passed. Exact-head CI and fresh Greptile/Paperclip review are pending. The prior head had 57 successful checks plus successful Snyk and Greptile 5/5; those results do not qualify this revision. - Environment: macOS arm64, Node 26.4.0, locked pnpm 9.15.4 dependencies, controlled remote command boundary and real Rust provider executor. No paid provider calls or hosted retries. - This is a bounded prevention change. It does not deliver historical provider artifacts to a replacement sandbox or replacement controller. Missing saved receipts or unavailable old artifacts remain a separate cold-recovery gap. No full hosted recovery is claimed. ## Risks - A continuation now requires its exact recorded pack when it has a durable binding. A newer installed pack cannot substitute for missing historical bytes. - Fresh launches, other providers and explicit continuity replacement keep their existing paths. Current ownership, company, model, permission and receipt binding checks remain in effect. - The repair does not relax the Rust launch-profile guard, replay prior tool effects, or authorize a new provider conversation. Existing published review receipts remain history. ## Model Used OpenAI Codex, GPT-6 family. The host does not expose the exact model ID or context window. Capabilities used: reasoning, code editing, shell execution, tests and authenticated read-only browser diagnostics. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
0e111c5075 |
fix(recovery): verify host preparation logs before continuation (#19580)
## Thinking Path > - Paperclip manages AI agents and the work assigned to them. > - A task can retain saved input after a failure during execution preparation. > - Retry must prove that the old attempt stopped and its files are saved before it admits a new turn. > - The host can write a skill announcement before it dispatches a provider. > - The historical preparation verifier requires an empty log and rejects this safe host output. > - This pull request verifies that exact small log against the saved skill selection and immutable log facts. > - Eligible saved input can continue once. Uncertain execution and current policy gates stay held. ## Linked Issues or Issue Description Refs: #19004, #19464, #19552. These repairs establish the existing preparation proof. They do not accept the host skill announcement described here. A search for the exact announcement found no duplicate repair. This is a bug fix within the existing recovery contract and does not duplicate a roadmap feature. **What happened?** A legacy preparation failed before provider dispatch with a host bootstrap certificate. Its remote lease stopped and its files were recovered. A newer preparation failed while that save was in progress. An accepted Retry then remained in execution recovery because its ancestor had one host skill log record. **Expected behavior** Admit one fresh turn for the existing saved user request after all execution, workspace and policy proofs pass. Preserve old outcomes and source records. **Steps to reproduce** 1. Create the real PostgreSQL fixture in `saved-bootstrap-continuation.test.ts`. It includes a recovered remote workspace, exact stop receipt, bootstrap ancestor, later failed preparation and saved user status queue. 2. Call `requestTaskRetry`, then call the real resume path concurrently. 3. On the baseline guard, the empty-log control dispatches. Adding only the host skill announcement leaves zero queued successors and the same execution-recovery wait. **Paperclip version or commit** Reproduced on master `65aee998a3`. The final branch rebases cleanly onto `c650f36007`. **Deployment mode** Source checkout with real embedded PostgreSQL and controlled log storage. The reported runtime is a server with remote sandbox workspaces. ## What Changed - Add a narrow verifier for one host skill-announcement record. It binds the complete bytes to the run log reference, digest, size, sequence, timestamp, output excerpt and recorded skill selection. Each announced skill must have a saved version; unrelated configured skills may also be selected. - Read at most 2,049 bytes with a two-second cancellation deadline. Missing, slow, changed, additional or provider output fails closed. - Use this proof only for historical bootstrap ancestors that already pass the no-provider, settlement, stop and saved-file checks. - Add real admission regressions for one successor, concurrent resume, duplicate Retry and delivery of the original saved comment. Add current-gate and storage-boundary negative cases. - Document the continuation contract. ## Verification - **RED:** On the unchanged baseline guard, the exact final fixture passes the empty-log control and fails the host-announcement case: zero successors instead of one. The saved comment and Retry remain deferred. - **GREEN:** 486/486 tests pass across `task-retry.test.ts`, `explicit-native-continuation.test.ts`, `saved-bootstrap-continuation.test.ts` and `verified-preparation-log.test.ts`. - The new cases cover provider-started evidence, provider events, a native coordinator, missing stop proof, missing skill selection, changed hash, unknown output, pending approval and operator Stop. - Read-boundary cases cover foreign references, changed bytes, truncation, extra records, malformed selection and cancellation. An unrelated recovery completes while another read is stalled. - After the rebase, the original 27 new cases passed again at head `be640292fe136232e3db0dc6938233f53b6ccb43`. The review fix adds three cases, and the full 486-test pass includes all 30 new cases. - The exact previously observed historical input also passes the production log verifier: one read, 240 bytes matching the stored size and SHA, the same timestamp and output sequence, and a recorded version for the announced skill. This uses captured bytes for transport and does not claim live hosted admission. - Full local server layered typecheck passes after building this checkout's Runner declarations and normal excluded-plugin SDK links. The typing correction also passes all 30 new cases and the actual captured historical log proof. The prior CI build/typecheck failure was TS7006 in the JSON parser; it is retained as failed evidence and corrected in this revision. - Module boundary and whitespace checks pass. Local verification selection requires broad CI because the execution contract changed; focused local suites are recorded above. [Required final-head CI](https://github.com/paperclipai/paperclip/actions/runs/42281590449) typecheck, build and tests pass at `c557cdc4dc57000cb478f9be0717675dcac7364e` (56 successful checks, no required check pending or failed). Greptile rates this final revision 5/5; the addressed thread is resolved. The advisory hosted Paperclip Review is paused with its request saved and has not supplied a final-head assessment. Its pending status is disclosed separately from required CI and Greptile. - Controlled fixtures exercise the admission and dispatch receipts. Hosted continuation still requires rollout and reassessment of the existing request. ## Risks This adds one bounded storage read to an active historical admission check. Failed or uncertain reads keep the existing hold. The host bootstrap certificate remains the primary no-provider proof. Log text alone cannot authorize continuation. No schema, retry budget, provider setting, account selection or scheduler changes. ## Model Used OpenAI GPT-6 family via Codex, with coding and tool use. The exact model ID and context window are unavailable in this session's metadata. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
96da284ed8 |
fix(workspaces): attach latest recovered files to Retry (#19542)
## Thinking Path > - Paperclip manages work performed by AI agents. > - A recovered run must retain its saved files when the user presses Retry. > - A prior fresh attempt can leave the task bound to a different workspace. > - Remote preparation restored older recovery attachments but omitted the latest canonical snapshot. > - The provider also received recovery paths only for fresh workspace attempts. > - This change attaches the latest verified snapshot and includes its paths in the normal continuation context. > - The selected workspace, its files, and execution admission rules remain intact. ## Linked Issues or Issue Description Refs #19479, #19340. **What happened?** Retry could start with an older recovery attachment while the newest saved files stayed in another workspace. A real database regression restored an older one-line log and omitted a newer verified twelve-line log. Workspace selection alone did not reveal this failure. **Expected behavior** The provider can find and read the newest verified saved files. Recovery preserves the selected workspace and existing files. Missing or invalid recovery proof stops preparation. **Steps to reproduce** 1. Recover a failed run into a canonical remote snapshot. 2. Retain a later selected workspace with an older recovery attachment. 3. Press Retry and inspect the actual files and provider continuation paths after preparation. **Paperclip version or commit** Reproduced on master `5d25117ffad69a93fb92026950b9acd9cec08f69` with synthetic data and local PostgreSQL 17. ## What Changed - Follow the admitted run's committed recovery ancestry as well as retained source IDs. - Import canonical snapshots as separate `.paperclip-recovered/` attachments. Reuse snapshot decryption, hash checks, and manifest validation. - Check the current writer, company, task, immutable committed snapshot receipt, and source access before import. Read-only historical proof remains valid when a later writer advances the source; execution admission proof is unchanged. - Preserve older attachments and later selected workspace files. Repeated imports retain the same path and preserve user edits. - Include only physically present recovery paths in ordinary provider continuation context, using the caller’s storage backend. - Document the contract and add actual-file regressions and negative evidence cases. ## Verification - Before the fix, normal Retry admission and remote preparation restored only the older one-line log. The newest twelve-line archive was independently readable but absent from the prepared workspace. - With the original context projection, full heartbeat workspace preparation restored the files but omitted their paths from provider context. The regression failed on the missing newest path. - All 28 focused real-database cases pass together at the final local revision (46.32s). An earlier run had one asynchronous comment-admission poll failure before workspace preparation; its bounded rerun passed, and the fresh final 28-case run also passed without timeout or assertion changes. These cover actual Retry admission, remote preparation, provider prompt rendering, missing and corrupt evidence, company and task scope, duplicate imports, and later workspace files. - A controlled local child reads the path from provider context and reports twelve lines with the expected first and last timestamps. It does not run a loop or write the log. - `node node_modules/vitest/vitest.mjs run server/src/services/workspace-recovery.test.ts -t 'restores the newest recovered tick bytes|canonical recovery attachments|deduplicates immutable canonical sidecars|caller.s storage backend|restores ordinary same-workspace Retry|admits one HTTP Retry|keeps verified copyback|prepares the .* task workspace|continues recovered preparation' --fsModuleCache` runs against a dedicated local PostgreSQL 17 test database. - Cold save and restore on a new allocation preserves user edits inside an existing sidecar. Both review findings failed before their fixes. Incomplete prepared or missing-snapshot references now stop preparation. - An upgrade regression reproduced valid legacy and canonical sidecar paths disappearing on same-allocation reuse. The fix rebuilds expected proofs from existing scoped receipts and checks the live destination, preserving user edits without copying files on that live allocation. The same actual-file case also saves the parent workspace, reserves it for a child, preserves both legacy and canonical paths, and refuses a revoked receipt. For related-task sidecars, retention verifies the selected workspace’s succeeded migration history, exact source run/task entry, and matching workspace and source-run markers. When legacy migration markers are absent, a verified committed snapshot of the selected workspace can establish the same exact provenance. It is read once per preparation with the existing decryption, hash, and manifest validators; the live workspace is never modified. Task relationships alone grant no access; fresh import stays scoped to the current task. Missing or mismatched migration history fails closed. The retained-path checks live in the attachment owner; remote-workspaces.ts remains below 1,000 lines. - Further regressions failed before the fix for deleted-sidecar context, caller-supplied storage, and a legitimately advanced source workspace. The fixed cases preserve edited sidecars and ordinary same-workspace Retry restore. - Real non-root regressions reproduced EACCES when cleaning read-only directories in import staging and snapshot verification. Permission-aware cleanup fixes both while preserving live 0555 directory modes and contents. - The complete remote file-transport suite passes: 123 tests, including six real-command provenance cases for absent, matching, foreign, symlinked, and hard-linked proof. - Server typechecks include the CI layered configuration, after building the current Runner TypeScript package and SDK dependency closure. `cd server && node scripts/typecheck-layers.mjs` passes all seven layers at the latest head (exit 0). - macOS arm64, Node 26.4.0, repository-pinned pnpm 9.15.4. The initial embedded PostgreSQL startup timed out before tests; the passing runs use a disposable PostgreSQL 17 container with temporary storage. No timeout was increased. - Required CI passes at `42495c7a73f3716240987fa403af7511fc9df9b9`: 55 successful checks, three expected skips, no failures; [aggregate verification](https://github.com/paperclipai/paperclip/actions/runs/42282134772/job/128121560893) passes. Greptile is 5/5 with zero unresolved threads. The separate hosted Paperclip Review remains queued and is disclosed as pending. This change has not been deployed; its affected hosted workflow remains held. ## Risks Canonical snapshot import adds bounded storage and archive work during active preparation. Upgrading an older related-task sidecar without migration proof may verify the selected workspace archive once; its temporary copy is removed and later runs reuse the recorded provenance. Missing objects, invalid receipts, or archive corruption stop preparation. Imports use separate directories and do not replace selected workspace files. Existing admission, Stop, budget, pause, and provider replay rules still apply. ## Model Used OpenAI Codex, GPT-6 family, with repository analysis, code editing, and tool execution. The runtime-specific model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
f513f0efd9 |
fix(recovery): admit certified process-loss preparation retries (#19552)
## Thinking Path > - Paperclip keeps task work durable across server restarts. > - Restart recovery records a certificate when a Runner preparation ended before provider work started. > - Automatic recovery already verifies that certificate together with saved files and stopped leases. > - Explicit Retry called a preparation helper that only reached this proof for status `interrupted`. > - A certified `failed/process_lost` source therefore stayed blocked after its files were recovered. > - This pull request makes the helper reuse the same existing proof. > - A queued user Retry can then admit one new turn while all current safety and policy gates remain effective. ## Linked Issues or Issue Description **What happened?** A task had a certified process-lost preparation, verified saved-workspace receipts, a destroyed source sandbox, and one or two later attempts refused before dispatch. Its accepted Retry request stayed deferred with `execution_recovery` and created no successor. The explicit admission helper excluded this source before it could use the preparation proof added in #19464. This also prevented the refused-attempt continuation path from #19466 from advancing. **Expected behavior** Explicit Retry must use the same positive pre-provider proof as restart recovery. It must preserve source history, refuse contradictory execution evidence, honor current policy gates, and admit at most one successor for the existing request. **Steps to reproduce** Persist a legacy preparing run as failed with `process_lost` and the host bootstrap certificate that provider work did not start. Record its canonical committed snapshot, recovery/finalization operations, clean workspace authority, and exact destroyed-lease receipt. Add one or two unstarted cancelled descendants with the dispatcher's execution-wait receipt. Accept one Retry request and resume it concurrently twice. Before this fix, no successor exists and the same request remains deferred. **Paperclip version or commit** Reproduced on master `a6c22806308e8a3c7b21145f80fb471756bb443d`. Related repairs: #19464 and #19466. ## What Changed - Reuse `isInterruptedNativePreparation` before the existing failed-preparation checks. The existing no-coordinator and historical-ancestor restrictions remain. - Add real-database regressions for one and two refused descendants, duplicate clicks, concurrent resume, continuation dispatch, preserved history, and unchanged recovery attempt counts. - Add late-gate checks for provider work, provider events, a newer native checkpoint, missing bootstrap proof, missing stop proof, a changed snapshot, formal approval, user Stop, agent/company pause, and a spent budget. - Clarify that explicit admission uses the existing certified preparation proof. ## Verification - Red: both real-database cases passed the saved-file, interrupted-preparation, and refused-dispatch proof checks, but the original successor-count assertion found zero queued runs instead of one. The accepted requests retained `execution_recovery`. - Green: the same cases admit one successor after two concurrent resume calls. Continuation dispatch resolves to the latest refused attempt. Source and cancelled-attempt history stay intact. - Complete relevant local suites: **456/456 passed** before rebase. Command from `server`: `../node_modules/.bin/vitest run --config vitest.config.ts --fsModuleCache src/services/task-retry.test.ts src/services/explicit-native-continuation.test.ts`. - Final head `9788382c019df57899af06e81211e7c5165e7f0d`: the thirteen new cases passed **13/13** again after rebase. Required CI and review remain pending. - Environment: macOS ARM64, Node 26, Vitest 5.0.3, disposable real PostgreSQL. Tests exercise production request, admission, and continuation services. No provider executes and no hosted task is changed. - Hosted recovery still requires rollout and verification through the original queued requests. Final reviewed head: `9788382c019df57899af06e81211e7c5165e7f0d`. [Required CI](https://github.com/paperclipai/paperclip/actions/runs/42273663624) passed (56 successful checks, four expected skips, none pending or failed). Greptile and Paperclip Review both rated this head 5/5 with no findings; all review threads are resolved. ## Risks The change accepts only the existing verified preparation contract. An error label alone is insufficient. Provider-started or contradictory state remains blocked. Saved-file, lease, approval, Stop, pause, and budget checks remain effective. No schema, retry limit, provider setting, account selection, or production scheduler changes. ## Model Used OpenAI GPT-6 family via Codex, with coding and tool use. The exact model ID and context window are unavailable in this session's metadata. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
c6eecac1e0 |
fix(heartbeat): keep a shutdown-suspended native recovery for the next boot (#19530)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Runner Full-Stack E2E workflow tests the Runner in Daytona sandboxes from a signed, content-addressed image > - Every change to an audited Runner input needs a new image before a Daytona cell can test it > - The image job took 10 to 16.5 minutes today, and a restart chaos campaign waits on it after each runner-side fix > - Measurement shows two serial waits: a full Rust dependency compile (about 5 of 6.6 minutes) and a 3.5 to 4 minute download of cached base layers that starts only after the builder stages finish > - There is also no way to build only the image: every dispatch reaches a paid gate that the operator must reject > - This pull request removes both serial waits and adds an `image_only` dispatch input > - The benefit is a shorter fix-to-Daytona-test loop, with every image security check unchanged ## Linked Issues or Issue Description **What happened?** The `Publish verified Daytona image` job is on the critical path of the control-plane restart chaos campaign. Builds today took 16.5 min (run 42240434011), 15 min (42253920727), 10.5 min (42259118276) and 16 min (42265637650). Run 42264611573 failed: its RunsOn runner lost communication with GitHub 7.6 minutes into the build step, during the Rust compile. GitHub has no log for that job. **Expected behavior** A Runner source change should not recompile the locked third-party Rust graph. Cached image layers should download while the builder stages compile. An operator should be able to publish only the image, without a paid gate. **Steps to reproduce** Read the BuildKit timeline in the logs of job 127970125495 (run 42240434011). See the stage table below. ## What Changed - `docker/daytona-runner/Dockerfile`: compile the locked Rust dependency graph in its own layer against placeholder targets. The layer depends only on `Cargo.toml`, `Cargo.lock` and the crate manifest. The step deletes the placeholder outputs and fingerprints, so the next step always compiles the real Runner sources. The final stage still receives only the stripped executable. - `docker/daytona-runner/Dockerfile`: move the content-ID / source-revision check above the two `COPY --from` steps. It does not depend on either builder stage, so BuildKit now downloads the cached base and CLI layers in parallel with the compile. Both copies use `COPY --link`. - `.github/workflows/runner-full-stack-e2e.yml`: add the boolean `image_only` dispatch input. `authorize`, `target_lock`, `catalog` and `daytona_image` run unchanged. Every other campaign job gets `&& inputs.image_only != true`. `authorize` rejects `image_only` together with TestOps inputs. - `tests/runner-e2e/workflow-image-only.test.ts` (new): pin the job classification, the top-level guard on each campaign job, the guard on every protected environment, and the TestOps rejection. - `tests/runner-e2e/README.md`: document the layer layout and the `image_only` command. ## Verification Measured on the same RunsOn `m7i-flex.xlarge` (4 vCPU, 16 GiB, native x64) runner class. "Before" is master job 127970125495 (run 42240434011). "After (cold)" is this branch, job 128077687214 (run 42271236140), with no dependency layer in the registry cache yet. Its paid gate was rejected. | Stage | Before | After (cold dependency cache) | |---|---|---| | Whole image job | 16.5 min | **11.6 min** | | Rust dependency graph | 311 s (inside one 399 s step) | 395 s (own layer; reused from cache after a master build) | | `paperclip-runner-core` compile | ~87 s | 65 s | | Cached base and CLI layer download | 236 s, **after** cargo finished | 257 s, **in parallel** with cargo | | Copy runnerd + provider pack | 20 s + 50 s | 9 s (`--link`) | | Provider pack stage | 143 s (parallel) | 152 s (parallel) | | Export + push | 82 s + 10 s | 81 s + 11 s | | Dependency layer size | n/a | `target` 852 MB + registry 333 MB (builder stage only) | With a warm dependency cache, a Runner source change compiles only the 65 s crate step. The critical path then becomes the ~4 min cached-layer download or the provider pack. The expected job time is about 7 to 8 min. I measure this after merge with an `image_only` dispatch (master first, to write the cache, then a Rust-only branch change). Disk: the instance starts with 38.76 GiB free. The dependency layer adds about 1.2 GB uncompressed to builder storage. The final image does not change. - `vitest run tests/runner-e2e/workflow-image-only.test.ts tests/runner-e2e/workflow-security.test.ts tests/runner-e2e/daytona-image.test.ts`: the new test and all image tests pass. Three tests in `workflow-security.test.ts` (`gates every provider-secret job with stable actor IDs`, `keeps provider credentials inside explicitly gated paid workflows`, `isolates private archive trust from workflows that execute target-branch code`) fail on `origin/master` without this change, because of `chat-catalog-testops-case.yml`. This pull request does not change them. - `node --test tests/runner-e2e/product-regression-testops.test.mjs`: 15/15 pass. - No local Docker build. The Dockerfile is measured with a branch-targeted CI dispatch whose paid gate was rejected. ## Risks - The dependency layer keeps the Rust `target` and registry trees in a builder-stage layer, and default-branch builds export it to the registry cache. This adds builder disk and cache upload when `Cargo.lock` changes. The build prints the size (`du -sk target /usr/local/cargo/registry`). - The dependency cache helps only after a default-branch build writes it. Branch builds read it but never write it. This is the existing cache policy. - `image_only` changes the workflow, so it works only after merge to master. - No check is removed. Lockfile verification, cosign sign and verify, the anonymous pull check, the provider-pack manifest match, the default-branch-only cache write and the actor allowlist are unchanged. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, GitHub CLI). Extended reasoning on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Devin FoleyandPaperclip
|
f0006c7783 |
fix(workspaces): contain inventory deadline aborts (#19543)
A capture deadline can expire while file identity validation awaits the filesystem. Recheck cancellation before creating the inventory read stream so the expired capture rejects without terminating the server. All 224 affected capture and archive tests pass, including real Node subprocess cancellation regressions. Final-head CI passes 54 checks, and Greptile scores 5/5 with no unresolved findings. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
07ddf9a5c2 |
fix(native): keep accepted results when runner close cannot prove suspension (#19525)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runs persist an accepted result first. The Runner then closes and suspends its session. > - Under heavy host load, the Runner close can fail to prove a durable suspension after the result is accepted. > - The executor then kept its finalization lease. The heartbeat finalizer got `native_finalization_lease_busy`, and the run ended `failed/adapter_failed`. > - The provider work was complete. Only Runner bookkeeping failed, so the run must not fail. > - This pull request hands the exact lease to finalization and completes the run from its accepted result. > - The benefit is that an accepted result stays accepted, independent of host load. ## Linked Issues or Issue Description **What happened?** A native ACPX Claude run asked a structured question. The control plane accepted its result (`Waiting for…`) and recorded `run.terminal` with `runTerminalState=succeeded`. The Runner close then took about 8.6 s. Its `runner.suspend` command ended `failed`, but the durable Runner state was `suspended` with a matching identity. The close threw `NativeSessionCloseUnrecoverableError`. The executor emitted `native.post_result_failure` (`category: runner_settlement_incomplete`, `resultAlreadyAccepted: true`) and rethrew `native_session_lease_lost`. Its finalization lease stayed held (its TTL is 20 minutes). `finalizeNativeRun` got `native_finalization_lease_busy`, and the run ended `failed/adapter_failed`. No restart was in progress. Host load average was about 40. **Expected behavior** After the control plane accepts a result, finalization bookkeeping must not fail the run. The run settles from its accepted result. Repair of the Runner checkpoint stays with the existing next-run admission rules. **Steps to reproduce** Run `pnpm exec vitest run src/services/heartbeat/runtime-execution.test.ts -t "accepted local result"` in `server/`. The test uses the real executor, the real control-plane result commit, and embedded PostgreSQL. The provider boundary accepts a result and then throws `NativeSessionCloseUnrecoverableError`. On the base revision, the finalization row keeps the executor lease (`phase: workspace_finalizing`) and the finalizer reports `lease_busy`. **Paperclip version or commit** Base `ff1d1dea5d`. The incident was on `819420a60a`. ## What Changed - `native-session-executor.ts`: after the rejected failure write proves an accepted result under the exact, unexpired lease, the executor releases that lease to finalization. It uses the same exact-authority predicates as the normal release. Before, the lease stayed held until its TTL expired. - `runtime-execution.ts`: for `runner_settlement_incomplete` on a local workspace (no workspace sync and no remote workspace authority), Paperclip records `workspace_finalize` as succeeded, finalizes the accepted result, and returns it to the ordinary completion path. - New `native-accepted-result-settlement.ts`: rebuilds the adapter result from the durable accepted result row, its terminal, the run's native session, and the accepted usage checkpoint. The result carries the closed `nativePostResultFailure` diagnostic. It contains no provider text. - Other post-result categories (for example `io_not_found`) keep their workspace-failure handling. They now also reach finalization at once, instead of after the lease TTL. - `doc/execution-semantics.md`: describes the settlement. ## Verification - Red, then green: `settles an accepted local result when runner close cannot prove suspension` and `keeps workspace failure handling for other accepted-result failures on a local workspace` in `server/src/services/heartbeat/runtime-execution.test.ts`. Both failed on the base revision with the executor lease still held. Both pass with this change. - Safety case `settles an accepted local result ... (workspace-check-failed)`: when the `workspace_finalize` check fails, the run keeps the existing failure path. Case `terminal-unreadable`: when the accepted result cannot be rebuilt, only a failed `workspace_finalize` record is written. - `pnpm exec vitest run src/services/heartbeat/runtime-execution.test.ts`: all pass (some cases time out at 30 s only under host load above 40; they pass on a narrow rerun). This includes the 14 existing post-result diagnostic and authority cases (expired, replaced owner, generation, attempt, boot, pid, process start, stopped, lock busy, and others). - `pnpm --filter @paperclipai/server typecheck`: pass. - Live cell `local-session-integrity.runner-acpx-claude.local.structured-question-restart-resume` (real ACPX Claude, local runner): 3 runs at host load average about 40. Runs 1 and 2 passed. Run 3 hit the fault naturally: the first run emitted `native.post_result_failure` (`runner_settlement_incomplete`) and now ended `run succeeded` (before this change it ended `failed/adapter_failed`). The cell then failed later, in the resume run after the answer: `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate; retirement_refused=target_not_sandbox_runner`. That is a separate follow-up (see Risks). ## Risks - A changed lease handoff. The executor releases the lease only after the rejected failure write proves an accepted result under the exact owner, attempt, controller generation, controller identity, and an unexpired lease. A stale or replaced owner does not release anything. A failed release is bounded and best effort. The old behavior then applies. - The run now completes as `succeeded` while the local Runner checkpoint is not proven suspended. The next run on the same session then meets the existing fence: the continuity-break retirement (`retireFinishedPerTurnRunnerdOwner`) accepts only sandbox Runners, so a local Runner is refused with `target_not_sandbox_runner` and its state root is quarantined. This PR does not change that fence. Extending the retirement to local Runners needs its own proof that the local Runner process stopped, and is a follow-up. - Settlement happens only when the accepted result and its terminal are rebuilt first, and the `workspace_finalize` record then succeeds. A failed branch or plugin workspace check keeps the failure path. - Synced and remote workspaces are not changed. They keep their copy-back retry path. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, test runs). Extended reasoning was on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Related: #19126 added the closed post-result diagnostic that this change uses. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
e3fb4bc73a |
fix(native): recover saved listener startup failures (#19479)
## Thinking Path > - Paperclip manages AI agents and their work. > - Native recovery must preserve saved work and respect current task authority. > - A closed local API listener can stop startup before a provider starts. > - #19462 gives new failures a bounded bootstrap retry. > - Older saved failures still have a generic terminal code and a recovery hold. > - This pull request verifies the old producer outcome and uses the existing safe replacement path. > - Eligible tasks continue once. Unknown provider outcomes and user holds remain blocked. ## Linked Issues or Issue Description Refs: #19462 **What happened?** A restart can leave an old listener startup failure in `terminal_failure` with `native_session_interrupted`. Startup recovery does not claim it after #19462. A completed file recovery can also retain its no-replay hold. **Expected behavior** Continue automatically when durable records prove that no provider started, the source stopped, and the current files were saved. Keep the existing attempt budget and normal admission checks. **Steps to reproduce** 1. Close the actual API listener after a fresh native coordinator claim. 2. Persist the old executor's generic failure. Save the workspace and confirm source termination. 3. Restart recovery. The baseline schedules no successor. The repair schedules one successor after all receipts pass. **Paperclip version or commit** Baseline: `0252421d5c`. Old failure producer: `2f35eb1080`. **Deployment mode** Built from source. Local macOS arm64, Node 26.4.0, real embedded PostgreSQL, independent filesystem execution. No external AI service is used by the controlled acceptance. ## What Changed - Recognize the exact saved local-listener guard through one proof predicate shared by discovery and the locked transaction. - Require a fresh first attempt, matching execution bindings, no prior provider lineage, no provider events or tool calls, no checkpoint or admission receipt, and no cancellation. - Reuse current snapshot, exact stop, source block, attempt budget, and successor idempotency checks. - Wait for completed source file-recovery receipts. Retire only that same source's verified preservation hold in the successor transaction. - Bound discovery to 100 candidates per pass, keep completed history quiet, and wake through existing transactional producer notifications. - Retain retries after transient database and final ownership-proof errors; advance the bounded cursor and force a rescan so a failing page cannot hide later work. - Let only the replacement duty discharge a proved-empty notification during drain through its existing read-only readiness check, once per notification generation. Keep real work and concurrent changes outstanding without running replacement during drain. - Wake replacement at canonical local-stop proof writes and the existing dispatch eligibility boundary; these hints grant no new execution authority. - Add database regressions, scheduler quietness checks, real controller handoff coverage, and an opt-in controlled Runner continuation. ## Verification Final production head: `0df41b2fef2644195d1224d36984af3827da3e3b`. Local environment: macOS arm64, Node 26.4.0, TypeScript 7.0.2, Vitest 5.0.3, frozen-lockfile dependencies. Tests execute freshly; `--fsModuleCache` reuses only content-validated module transforms. - Native safe replacement: 62/62 PASS on isolated PostgreSQL 17 (63.18 seconds), including clean per-suite database disposal. All 62 assertions also passed on embedded PostgreSQL 18 beta, but its Darwin cleanup hook timed out; that invocation is not counted as a pass. Docker's disk was full, so the disposable PostgreSQL 17 data used a size-limited RAM mount, with no host mounts. The owned container was removed afterward. - Actual stopped-process producer suite: 31/31 PASS. A real exited child produces the stop receipt, and its replacement notification settles only after the outer transaction commits. - Workspace and separate-process drain regressions: 33/33 PASS (75.90 seconds). - Shared coordinator and execution-control lifecycle checks: 39/39 PASS (16.43 seconds). Pending work, concurrent changes, cold startup, shutdown, and passive drain boundaries remain covered. - Actual suppressed Cloud routine/drain workflow: PASS (34.64 seconds), using the unchanged test and timeouts. Its matching local RED reproduced CI's 462-query, zero-pending-query, one-idle-hold timeout after an admitted write committed during drain (69.46 seconds). - Actual historical controller handoff: PASS (48.40 seconds). A separate controller loads the original executor from `2f35eb1080`, writes its generic listener failure, and exits. Recovery at the final head admits exactly one successor. The real Rust Runner and controlled Codex process complete governed `paperclip_finish`; attempt two commits and the saved file remains intact. Only the child loads the old artifact, and both isolated artifacts are restored afterward. Rust source subtree `77961b70f177ec3c38d11abcaec92fd3d0e96ec0` matches the built binary. No external AI service is used. - Seven corrected existing fixtures pass: transactional stop-proof calls (5 cases) and notification inventory (2 cases). Their earlier CI failures exposed fixture calls outside the documented transaction and an outdated expected queue set. - Server typecheck: PASS. Red/green proofs also cover the original saved failure on baseline `0252421d5c` (zero successors instead of one), held-history quietness, bounded forward scanning through 100 parked rows and concurrent wakes, real snapshot/cleanup producer notifications, actual issue-row lock timeout, and task/history eligibility notifications. An obsolete prior-run question can be skipped to request another proof check; actual skip while the task is paused or the source owns provider tool activity schedules no successor. The failed-page regression starts with 25 persistent verifier errors and a healthy candidate 26. Baseline RED repeats the first page. Repair GREEN advances the healthy candidate, revisits the original failures after their fault clears, creates all 26 successors exactly once, and becomes quiet. Greptile is SUCCESS, 5/5 on the exact final head (2026-11-06 17:09:21 UTC). Final-head CI is GREEN: 56 successful checks, three intentional skips, one neutral result, and no pending or failing checks. [Attempt two and results](https://github.com/paperclipai/paperclip/actions/runs/42258352876/attempts/2) include passing build, typecheck, Rust, native server, package, browser E2E, serialized server, general server/chat tests, and final verification. Five jobs in the first attempt were interrupted together at 17:13:12 UTC; the native-server log records an explicit runner shutdown signal. The single infrastructure retry reran only those five jobs and dependent verification, preserving the 49 successful checks on the unchanged head. No timeout was weakened, and setup failures or interrupted invocations are not counted as passes. Focused commands from the repository root: ```sh ./server/node_modules/.bin/vitest run --project @paperclipai/server server/src/services/remote-workspaces.integration.test.ts server/src/services/native-runtime/native-runner-restart-preparation.integration.test.ts -t 'native safe-replacement decision|reconciles the drained remote backend' --fsModuleCache --reporter=dot ./server/node_modules/.bin/vitest run --project @paperclipai/server server/src/__tests__/execution-control-work.test.ts server/src/__tests__/delivery-work-coordinator.test.ts -t 'execution-control lifecycle|work coordinator' --fsModuleCache --reporter=dot node node_modules/typescript/bin/tsc -p server/tsconfig.json --noEmit --pretty false ``` The historical acceptance sets `PAPERCLIP_TEST_HISTORICAL_LISTENER=1` and uses an original-executor child artifact with a `try/finally` restore. Normal regressions require no Rust binary. ## Risks - Compatibility is deliberately limited to positively proved fresh remote startup failures. Other old ambiguous failures remain held. - Existing provider work, a changed task block, incomplete preservation, or revoked authority must prevent execution. Normal retry admission remains in control. - This PR does not deploy the change or mutate a hosted task. Hosted recovery still requires the repaired revision and all current proof gates. ## Model Used OpenAI GPT-6 through Codex, with reasoning, tool use, and code execution. The exact backend model identifier and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
a0cba70f18 |
fix(runner): continue restart-interrupted ACPX and OpenCode turns (#19501)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs keep their provider conversation when the control plane restarts > - A local Codex turn lost in a restart already continues once in the same conversation (#15297) > - ACPX (Claude and the other ACPX profiles) and OpenCode took a different path: the runner closed the lost turn as a provider failure > - The task then blocked with "Provider turn failed", and the user's queued follow-up stayed unread > - A restart must not need a user action when continuing is safe > - This pull request gives ACPX and OpenCode the same recoverable restart-loss cause and the same single continuation as Codex > - The benefit is that work interrupted by a controller restart finishes on its own ## Linked Issues or Issue Description Refs #15297 (Codex continuation), #19399 (restart-interrupted turn hook), #19471 (harness fix used for the live reruns). **What happened?** After a control-plane restart during an active native ACPX Claude turn, the task ended `blocked` with `native_provider_terminal_failed: Provider turn failed`. The queued follow-up comment was not consumed. No continuation started. The next run then failed with `native_session_cleanup_quarantined`. A native OpenCode run killed mid-turn ended the same way (`turn.failed` with `error: null`). **Expected behavior** When the old provider process is gone and the conversation can be restored, continue the task once in the same conversation. When that is not safe, record an honest restart cause instead of a generic provider failure. **Steps to reproduce** 1. Start a native ACPX Claude task on a local runner. 2. Submit a follow-up comment while the turn is active. 3. Restart the control plane while the turn is still running. 4. Run `pnpm test:e2e:runner -- --max-automatic-retries 0 --id everyday-workflows.runner-acpx-claude.local.recover-controller`. **Root cause** - `runner-core/src/acpx_provider_backend.rs` `restore_session_if_needed`: a saved `turn_active` state always closed the session. It recorded `turn.failed` with only a top-level `code: acpx_active_turn_recovery_closed` and no `error` object. It also emitted `run.terminal`. - The runnerd transport forwards only `error`, so the controller saw `error: null` and reported generic `provider_turn_failed`. - `runner-core/src/provider_backend.rs` gave the recoverable `provider_turn_lost_on_restore` cause only to `provider_label == "codex"`. OpenCode shares that backend. - `nativeRestartInterruptedTurnId` accepted only `codex_app_server`. The runtime continuation gate accepted only `provider.kind === "codex"`. - Two follow-on defects appeared once the continuation ran live: - The restored ACPX provider rejected the controller's result for the lost turn's open semantic call. That call never settled. - The restored provider's MCP request ids restart. The controller keys semantic calls by `hash(runId, callId)`. The continuation's first Paperclip tool call reused an earlier id, was treated as already answered, and hung. After the result was accepted, `runner_settlement_incomplete` failed the run. ## What Changed - ACPX runner: for an active turn lost to a runner restart, prove that the original provider lifetime lost its fence quorum. Then restore the same conversation and record `turn.failed` with `error: { code: "provider_turn_lost_on_restore", recoverable: true }`. Keep the run open for the controller's continuation. An accepted result, an unanswered runtime request, or an active goal keeps the old close behavior. So do an unproven lifetime fence and a failed restore. - ACPX runner fallback close: add a structured `error` (`acpx_active_turn_recovery_closed`, not recoverable, with an honest restart message), so the cause survives the transport. - ACPX runner: track open semantic calls durably. On restore, retire the lost turn's open calls. Acknowledge their controller results as `retired` and do not send them to the new provider. - ACPX sidecar: when the sidecar restores an existing session, prefix forwarded semantic `callId`s with a per-process namespace (`r<hex>:<id>`). Fresh sessions keep raw ids. - OpenCode runner: give OpenCode the same recoverable cause and open run as Codex (`restart_lost_turn_is_continuable`). Cursor Cloud stays excluded because its agent runs remotely. - Runner TS: `nativeRestartInterruptedTurnId` accepts `acpx_runtime` and `opencode_server`. The runtime continuation gate accepts `acpx` and `opencode`. All other Codex safety rules are unchanged: one continuation, admitted dead-runner restart only, no compaction, goal, semantic result, or pending runtime request. - Docs: `doc/execution-semantics.md` and `doc/run-log-events.md`. ## Verification Red, then green, at each layer: - Rust `tests/acpx_warm_attachment.rs` `restores_the_conversation_after_losing_an_active_turn_to_a_runner_restart`. Red: status `closed`. Then red on the lost call's result: "tool result does not match a pending provider call". Green after the fix. - Rust `tests/native_provider_backend.rs` `restart_lost_opencode_turn_records_a_recoverable_cause`, with the qualified OpenCode launch fixture. Red: `error` was `null`. Green after the fix. - Rust unit test `restart_lost_turns_continue_only_for_locally_restored_providers` (Codex, OpenCode yes; Cursor Cloud no). - TS `native-session-runtime.test.ts` "continues a restart-lost acpx_runtime / opencode_server turn once". Red: `NativeProviderTerminalFailure provider_turn_lost_on_restore`. Green: exactly one continuation, which does not resend the task. - TS `acpx-runtime-sidecar.test.ts`: a restored provider's call ids are namespaced and resolve. - Server `native-runner-restart-recovery.integration.test.ts`: a checkpointed `acpx_runtime` or `opencode_server` restart interruption is claimed as `resume_dead_runner`. Red without the driver gate. Green with it. - Two `native_provider_backend.rs` tests asserted the old close-on-restore result. They now assert the new contract. Their real invariant still holds: the delivered response is never replayed. Suites run locally on the final revision: - runner-core: `cargo test --no-fail-fast acpx`, `--test native_provider_backend` (38), `--test acpx_warm_attachment`, `--test codex_provider restart_lost` - runner TS: `native-session-runtime`, `runnerd-restart-continuation`, Codex driver recovery, contracts, and `acpx-runtime-sidecar` (all pass) - server: `native-runner-restart-recovery.integration.test.ts` (80 passed) - `pnpm --filter "@paperclipai/paperclip-runner..." build` (all generated-contract checks pass) Live runner E2E (local, real Claude via ACPX, macOS arm64): - `everyday-workflows.runner-acpx-claude.local.recover-controller`: PASS twice in a row on the final runner code. The restart crossed an active turn. The task finished `done`, the queued follow-up was consumed, the ZIP download was verified, and there were no failure codes. - `stock-harness.runner-acpx-claude.local.continuity-restart` regression: PASS. - Earlier failing reruns are kept as evidence of the follow-on defects: one harness `kill EPERM` while stopping the server (unrelated to this change), one hang on the colliding call id, and one `runner_settlement_incomplete`. ## Risks - A continuation can run more work in the restored conversation. Its prompt requires inspecting state before repeating an uncertain action, as for Codex. - The ACPX restore waits up to 5 s for the original provider lifetime fence before it falls back to the old close. - The ACPX provider state file gains two fields with defaults. Older saved states still load. A downgraded runner cannot read state that a newer runner wrote with these fields. - Restored ACPX sessions now forward prefixed call ids. Only fresh sessions keep raw ids. Live proof covers Claude. Codex-ACP, Pi, Cursor, Copilot, Grok, Kimi, and Antigravity share this path, but this PR proves them only by unit tests and code review. - OpenCode is covered by the runner fixture and gate tests. The live OpenCode restart-chaos cell was not rerun here. - `runner-core` `unconfirmed_suspension_state_becomes_recoverable_only_after_cleanup_and_suspend` is flaky when run in parallel. It also fails with the new path disabled, so it predates this change. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use, code execution, and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
c6573962f8 |
fix(workspaces): recover full sandboxes through read-only export (#19465)
## Thinking Path > - Paperclip preserves agent work across failed runs and server restarts. > - A remote workspace must have a durable save before its source can be released. > - Portable capture normally writes a Git mirror and a full archive inside the sandbox. > - A full sandbox can prevent capture before the provider's streaming download starts. > - Retrying that write cannot create the required free space. > - This change transfers a checked standalone source and seals its archive on the controller. > - The source remains retained unless the existing durable save completes. ## Linked Issues or Issue Description Related: #18644 reduced temporary capture space. #18172 added the provider's disk-free directory transfer. This repair connects portable capture to that transfer when the archive itself cannot be created. **What happened?** Workspace capture returns `remote_workspace_sandbox_disk_full` after provider work has finished. File recovery repeats the same sandbox write. **Expected behavior** Recover a stable standalone workspace without requiring another full archive on the source disk. Preserve staged changes, untracked work, confined links and the original source. **Steps to reproduce** 1. Create a standalone Git repository with a commit, staged changes and an untracked result. 2. Use the existing real-command transport fixture. Inject ENOSPC into the sandbox helper write or archive write. 3. Run portable capture. Before this change it rejects the helper failure. 4. With the change, restore the saved archive after removing the fixture source. Check the commit, staged content, untracked result and Git connectivity. ## What Changed - Attempt one controller-side capture after an exact sandbox disk-full failure. - Read a source inventory without installing a helper. Use the existing provider bulk transfer with confined links and runtime exclusions. - Compare the transferred inventory with source fingerprints before and after sealing. - Reject copied Git pointers, alternates and administrative symlinks. Parse config as data without includes; allow only portable keys before controller Git normalization. - Preserve socket exclusions. Retain the source and remove incomplete controller staging after failure. - Update execution semantics and focused regression coverage. No database migration or new execution authority. ## Verification - TDD red on base `2f35eb1080`: the new full-sandbox test failed with `remote_workspace_sandbox_disk_full` before production edits. - Green: 202 transport and archive tests passed before the additional Git configuration guards. At head `8541086cb5`, all 116 transport tests and the PostgreSQL durability acceptance pass. The final `e24e173ec9` exclusion guard passes those 116 cases plus its new Git-history regression across the full run and focused rerun. Environment: macOS arm64, Node 26.4.0; real Node/Git commands and filesystem transfer. - Review regressions were also red before their fixes: an internal Git config symlink bypassed the check, and `core.attributesFile` could select a controller path. Git administrative symlinks and nonportable config now fail before normalization. The focused safety selection passes 11 cases. - Cases cover helper and archive ENOSPC, commit/staged/untracked continuity after source removal, confined links, sockets, changed source, corrupt or extra copied files, unsafe Git metadata, retained diagnostic text and failed download. - Real PostgreSQL acceptance passes: normal workspace save recovers one sandbox ENOSPC, commits the encrypted snapshot, clears its writer, loses the original allocation, then independently reloads the saved result twice. The first acceptance attempt used an incorrect expected status (`available`); the documented state is `committed`. No production change was needed for that correction. - Review regressions for untracked Python/Node environments and excessive Git roots failed before their fixes. Dependency exclusions now match normal capture. Controller parsing starts only after whole-tree/repository budgets and layout checks; refspec/settings budgets and a shared two-minute verification/normalization deadline bound further work. The excessive-root test verifies zero Git subprocesses. - Real provider-boundary red/green: Daytona direct tar and capacity-fallback PTY transfers first dropped `._result.txt`. The new optional mapping contract preserves all user names and excludes only current transfer scratch. Kubernetes implements the same contract. Explicit caller exclusions and archive validation remain in force. - Standalone Linux qualification on Node 24.22.2/Debian bookworm: fresh shared/SDK build; Daytona and Kubernetes typecheck and build; Daytona file-sync/capacity/diagnostics 68/68 pass; Kubernetes file-sync 31/31 pass, including the actual `/proc` case. Qualification was refreshed again on the final `be6f7feacc` source tree: current shared/SDK rebuilt; all 815 staged source/config/manifest hashes match the delivered tree before and after testing. An initial container run as root bypassed a read-only file test; the same source passes as uid 1000. Installing zstd enabled eight initially skipped cases. - A further review regression proved `.git/pyvenv.cfg` could exclude Git history as a dependency. The inventory now handles `.git` before dependency filtering, matching normal capture. The full capture/restore test verifies the original commit and saved work. The first green attempt incorrectly compared command-result object identity; correcting the test to compare stdout passed with no production change. - Further red/green regressions cover read-only private-copy cleanup, a regular file replaced by a FIFO before open, case aliases of Git administrative metadata, and permission preservation under restrictive umask. Nonblocking no-follow file handles verify identity before and after reading. Cleanup changes only private controller directories. Provider extraction preserves ordinary permissions, refuses special mode bits before writing, and does not adopt source ownership, ACLs, attributes or flags. Final macOS Daytona tests pass 65/65; the final Linux qualification above was repeated on these exact provider sources. - Focused strict TypeScript compilation of the inventory and archive modules passes. - The first sandboxed archive run failed three Unix socket fixtures with EPERM. The permitted run outside that filesystem sandbox passed all tests. - Initial full local server typecheck could not resolve the unbuilt Runner package and produced cascading errors. Exact-head `e24e173ec9` CI typecheck, build, Runner and browser checks pass. That CI attempt failed two server shards: the unchanged retained-cleanup backlog test timed out at 15 seconds and passes locally unchanged; ten cache-reclaim fixture oracles failed because directory fallback consumed their sealed-archive counters/faults. Final commit `caed4e4f83` isolates the intended unavailable-directory boundary while preserving archive counts, recovery status, integrity and no-second-reclamation assertions. Those ten failures reproduced locally before the fixture edit. The full affected file passes 394/395; its sole other HTTP socket-hang-up case passes unchanged in an isolated rerun. This is combined coverage of all 395 cases, not a single complete green run. The final branch integrates master `53645a0104` at `98ea07225592930cbb0d18bcf250bc64f6bb13b4`, preserving upstream workspace phase timings. After integration: 117 transport cases, the PostgreSQL durability acceptance, the existing object-storage/timing regression, and 12 affected cache cases pass. The expanded recovery file now has 421 cases. Final-head `be6f7feacc45c1e32a26a2417e973f1e3752fd8d` CI passes all 55 applicable checks (three skipped, one neutral review), including the expanded recovery suite, typecheck, builds and Runner coverage. [CI result](https://github.com/paperclipai/paperclip/actions/runs/42251345554). - Provider exclusion regressions were red through four real Linux transfer paths: Daytona direct archive, direct safe-link repair, capacity PTY repair, and Kubernetes streaming. A root socket and nested regular file share a basename; literal glob-character files/subtrees also survive. Preserve mode now anchors tar exclusions as literal root-relative paths and escapes `find -path` pruning. Default callers keep glob semantics. Receipt-count regressions were also red before correcting preserved-output counting. All 31 Kubernetes and 68 Daytona tests, builds and typechecks pass on the final source with 815 matching inputs. - The final semantics documentation explicitly records special mode bits, the stricter config allowlist (including innocuous unlisted keys), and provider qualification limited to Daytona/Kubernetes; these address the optional current-head review observations without broadening controller Git authority. - No Daytona credential is available in this session. Provider transfer is a local filesystem boundary in these tests. Hosted task recovery and rollout remain unverified. ## Risks - Recovery reads the source repeatedly and transfers the standalone tree. It can take longer than normal capture. - Linked worktrees, initialized submodules, Git administrative symlinks, nonportable Git configuration, special mode bits and alternate object stores remain retained if the normal capture cannot finish. This fallback does not follow their pointers onto the controller. - A provider without usable directory export, a changed source, or insufficient controller space cannot produce a receipt. The original source stays retained. - The server, SDK and affected provider plugins must be built from this revision together. An older provider can fail inventory verification on names it omits; it cannot publish an incomplete save. No merge, publication or deployment is included. ## Model Used OpenAI GPT-6 family through Codex, with reasoning, code editing, shell tools and automated tests. The exact serving model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub references) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
7840d894fa |
fix: retain files after accepted remote copy-back exhausts retries (#19491)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Remote workspaces keep unsaved files bound to the run that wrote them. > - An accepted result can exhaust its copy-back retries before those files are saved. > - Failed-workspace preservation rejects every accepted result, so a stopped writer can keep the workspace held after a newer setup attempt fails. > - This pull request lets the existing owner retain files from that exact stopped remote writer. > - File recovery can then release the hold without applying the old result or repeating its provider work. ## Linked Issues or Issue Description **What happened?** A failed native remote writer had an accepted result and exhausted copy-back retries. Its sandbox was stopped with a valid retained-source receipt. A newer attempt failed on the writer's unsaved workspace. Task Retry could not register the missing file-recovery obligation because preservation returned as soon as it found the old accepted result. **Expected behavior** Task Retry must let the existing recovery owner save the exact retained files. The accepted result, failed attempts and current task must stay intact. The old provider work must not run again. **Steps to reproduce** 1. Finish native work in a remote task workspace and accept its completion result. 2. Exhaust copy-back retries, stop and retain that exact sandbox, and leave the workspace dirty. 3. Let a newer setup attempt fail on the old writer's hold. 4. Use ordinary task Retry. Before this fix, the missing file obligation is never recorded. **Paperclip version or commit** Reproduced against master at `51a65af9b81f3c60f8210047e9a3eeaada7bc155` with isolated PostgreSQL and real files. Related: #19385 preserves accepted copy-back interrupted by Stop. #19422 admits saved input after verified recovery. #19464 and #19466 repair preparation and Retry continuation. This change covers an older remote writer whose accepted copy-back has exhausted its retries. ## What Changed - Admit only terminal, exhausted accepted copy-back on a failed remote writer with a finished run and no live controller or finalizer. - Require the exact stop-and-retain receipt, accepted completion contract and current dirty workspace binding. Keep native-sync sources on their existing export path. - Use the existing file-preservation recorder and worker. Add no new recovery worker, schema, polling or replay path. - Cover bound and orphaned entry points, reusable leases, changed proof and the ordinary HTTP Retry journey. Document the recovery behavior. ## Verification - Causal baseline: the corrected bound-writer and orphaned-lease tests both fail without the source change and pass with it. - Assembled acceptance: authenticated HTTP Retry records the old writer's missing obligation; the real recovery service saves its files; concurrent continuation calls queue exactly one successor for the newer failed setup. PostgreSQL, filesystem, tar, Git and the API/service path are real. Provider transport is controlled, and no model turn is dispatched. - `pnpm check:module-boundaries`, `pnpm -r typecheck`, `pnpm build` and `git diff --check` pass. - Four full affected recovery suites pass: **659 tests**. Command: `pnpm --dir server exec vitest run --config vitest.config.ts src/services/workspace-recovery.test.ts src/services/native-runtime/native-failed-workspace-recovery.test.ts src/services/native-runtime/native-cancelled-workspace-preservation.test.ts src/services/native-runtime/native-workspace-export-retry.test.ts`. This includes the final 16 new regression, refusal and assembled journey cases. - `pnpm verify --explain` completed. It selected broad transitive coverage; this is planning evidence only. Hosted CI will supply the remaining broad coverage. - Dependency cost: extra indexed source/result reads run only for this exact terminal retained-result path, inside the existing bounded transaction. Existing notifications activate the existing recovery owner. No idle scan is added. - Fixture failures are accounted for: the first fixture used an active-only stop helper on a released lease. The corrected fixture produced the causal baseline above. One negative fixture tried to remove a contract while an accepted result referenced it and failed the database foreign key. That impossible fixture was removed; the positive journey passed in the same run. Neither fixture failure is reported as a product regression or a passing full run. - Hosted CI is fully green at the reviewed commit: 54 successful checks and 3 intentional skips, with no incomplete or failed checks. Greptile is 5/5 with no unresolved threads. - Hosted rollout and a live task continuation remain pending. Local API admission does not prove a deployed model run. ## Risks This expands file preservation for a narrow terminal failure. Wrong source identity could retain unrelated files; unchanged writer, epoch, location, provider lease, stop receipt and accepted contract checks fail closed. Active ownership, unsafe archives, rejected results and native-sync sources stay on their existing paths. Existing recovery owns physical source locking, copy verification and bounded retries. It preserves unknown action outcomes and leaves the accepted result uncommitted. There is no migration or configuration change. ## Model Used OpenAI Codex, `gpt-6.1-sol`, xhigh reasoning effort. Context window size is unavailable. Used repository tools, isolated tests and code execution. No delegated agents. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
51a65af9b8 |
fix(recovery): continue preserved preparation after process loss (#19464)
## Thinking Path > - Paperclip manages AI agents and their execution state. > - A server restart can stop workspace preparation before provider execution. > - Workspace recovery must preserve files before the task can continue. > - Recovery handled interrupted preparations but skipped failed process-loss records with the same no-provider certificate. > - A withheld first retry could also hide the remaining retry budget. > - This change resumes certified preparations through normal admission after file recovery. ## Linked Issues or Issue Description **What happened?** A process-loss attempt remained failed after workspace recovery completed. Its first retry had been cancelled before dispatch by the workspace hold. The task had no running attempt or automatic continuation. **Expected behavior** Continue once after the server proves that provider work did not start, file preservation completed, and the exact leases stopped. Retain the existing failure budget and all task admission gates. **Steps to reproduce** 1. Stop a Runner preparation before native runtime selection. Record the restart recovery bootstrap certificate. 2. Retain the workspace and withhold the first automatic retry before dispatch. 3. Complete file preservation and restart the recovery worker. 4. Observe that the old code schedules no successor. The regression test reproduces this with PostgreSQL and real filesystem operations. **Paperclip version or commit** Reproduced against master at `2f35eb1080`. Related prior work: #19422 continues saved user input after verified copy-back. This change handles automatic continuation of certified process-loss preparation. ## What Changed - Accept failed process-loss preparation only with the durable no-provider bootstrap certificate and existing execution, lease and file checks. - Discover and settle the recovered source through the existing preparation recovery path. - Continue from a verified withheld retry without rewriting its history or resetting its failure count. Adopt it only as the sole retry child; multiple-child history reuses the newest existing child deterministically. - Exercise automatic dispatch, preserved-file readback, one provider turn, accepted result and task completion in the local native-session fixture. - Keep a durable continuation receipt after file-hold settlement until a successor is recorded. Known gates wait for existing eligibility events. Unresolved delivery keeps its shared recovery deadline. - Document the recovery and retry-budget rules. ## Verification - Red: both new PostgreSQL/filesystem regressions failed with zero scheduled successors where one was required. - Green: 54 focused recovery, automatic-dispatch, interruption, multiple-child history and admission safety cases pass on `13c27c2c2f`. The full PostgreSQL admission adapter suite passes all 83 tests; server source declaration typecheck also passes. Local environment: Darwin arm64, Node 26.4.0, PostgreSQL 18.1, Vitest 5.0.3. - Additional red/green: interruption after committed settlement, pause/resume, failed enqueue after cursor advancement, and absence of an earlier retry all failed before their repairs. Restart before enqueue, restart after the successor claim, and pause/resume now preserve one continuation. A rolled-back eligibility change cannot start execution. Deferred and delivered records have no retry deadline. - Local acceptance uses the real heartbeat dispatcher, isolated files and PostgreSQL, with a simulated native provider. It reads the preserved file, writes continued work, accepts the structured result and completes the task once. - Negative cases retain no-certificate, provider-evidence, live-owner, Stop, missing-stop-receipt, wrong-snapshot, human-block, pause, reassignment, approval and exhaustion fences. - CI fixture regression: 4 error-propagation/concurrent-resolution cases pass after adding the new obligation read to their transaction stubs; existing assertions remain unchanged. - Existing idle-drain and notification contract regressions: 4 tests pass after restricting recovery wakes to actual durable obligations. Late approvals block scheduled, queued and final dispatch. - Additional existing retry-scheduling suite: 62 cases pass locally. The pre-provider quota case exceeds its existing 5-second terminal wait on this loaded Darwin host, including when tested alone and with the preceding retry implementation. It remains a local timing limitation; the assertion and timeout were not changed. All Linux CI server shards pass on the combined head, including this unchanged suite. - Existing run-control, legacy-recovery and shared execution-control lifecycle suites: 128 tests pass with the review repair. Required CI supplies final-revision broad coverage. - Real-Codex acceptance was not run: automatic approval review rejected authenticated provider execution because it could incur cost/external effects and required explicit approval. No provider call ran; the controlled-provider acceptance covers the causal restart and durable scheduling behavior. - Hosted acceptance and recovery of existing tasks require rollout. They have not been claimed from the local provider fixture. - Final-head review: Greptile 5/5 on `13c27c2c2f`, no outstanding findings. Another contributor added the multiple-child history safety repair in `13c27c2c2f` while CI was running. The delta was inspected and the 54 focused recovery cases rerun successfully. Required CI passes on that exact combined head: https://github.com/paperclipai/paperclip/actions/runs/42241794918. All 55 successful checks include aggregate `ci / verify`, typecheck, build, canary, all server and E2E shards, and Runner package/Rust/static checks. The separate Paperclip Review service check remains queued waiting for its assigned agent; no validated assessment has been returned. ## Risks A malformed preparation certificate or a false no-execution receipt could permit duplicate work. The existing checks still reject provider execution records, native coordinators, live controllers, unsettled leases and unverified files. The pending continuation must survive a restart without polling known admission blockers. A withheld retry retains its original row and consumes its remaining failure budget. Normal admission checks current ownership, approvals, pause and budgets. ## Model Used OpenAI GPT-6 family via Codex. The exact model ID and context window are unavailable in session metadata. Used reasoning, code editing and tool execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal task id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Devin Foley <devin@paperclip.ing> |
||
|
DottaandPaperclip
|
06c0566a08 |
fix: make task Retry work after a refused automatic attempt (#19466)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Task Retry stores a durable request before it starts a new run. > - An automatic attempt can stop at the execution gate before a provider starts. > - That cancelled attempt becomes the source of the next Retry click. > - Admission rejected that source, or chose an older run that did not match the stored Retry authority. > - This pull request verifies the refused attempt and keeps it as the exact authority source. > - The user can start one new turn after the older execution holds pass their existing checks. ## Linked Issues or Issue Description **What happened?** Retry can remain deferred when the latest run was cancelled with `execution_reconciliation_required` before dispatch. An older failed preparation has an execution recovery hold. The Retry receipt names the cancelled child, but admission only accepts a failed preparation as an extra source. It can also choose the older run as `previousRunId`, which fails the exact source check at dispatch. **Expected behavior** Retry starts one new turn when the latest attempt has a verified pre-dispatch refusal and every older execution hold is ready. Provider activity, retained execution resources, missing receipts, or a live earlier process keep the hold. **Steps to reproduce** 1. Seed a failed native preparation and its recovery hold. 2. Let a later automatic attempt be cancelled by the stale queued execution gate before dispatch. 3. Click Retry against that latest cancelled attempt. 4. Before this change, continuation admission returns no receipt. After this change, one successor starts with authority bound to the clicked attempt. **Paperclip version or commit** Reproduced on `2f35eb1080`. Rebased onto `6cafcaf89b` for submission. **Deployment mode** Local source checkout with embedded PostgreSQL and the native Runner on macOS arm64. Refs #15383 and #16939. Those changes cover legacy conversation cancellation and queue promotion. This change covers the exact source of a task Retry receipt after a gate refused its automatic child. ## What Changed - Reuse the existing database-backed pre-dispatch refusal proof when admitting the exact Retry source. - Keep the clicked source as `previousRunId`. Continue to check every older execution hold. - Add database regressions for safe refusal, unsafe evidence, concurrent dispatch, and duplicate clicks. - Add an opt-in acceptance test that drives the authenticated Retry route through a real Codex turn and task finalization. - Document the admission contract. ## Verification - TDD: the new database regression returned `null` on the original code. It passes with the repair. - Final head `c43a84b1bf`: 487 tests pass across task Retry, explicit native continuation, and execution continuation. Server TypeScript and module boundary checks pass. - Live acceptance: `PAPERCLIP_LIVE_REFUSED_RETRY=true PAPERCLIP_RUNNER_BINARY=<current-source-runnerd> pnpm --dir server exec vitest run --config vitest.config.ts src/services/native-runtime/refused-task-retry.live.test.ts`. - The real HTTP request returned 202 on the final rebased head `c43a84b1bf`. Codex `gpt-6.1-sol` with high effort produced the requested file. The task became done, both task locks cleared, and finalization committed. A repeated click reused the same receipt and did not create another run. The source file stayed unchanged. - Server TypeScript check and module boundary check. - `pnpm verify --explain` returned 2 because the execution contract doc requires broad CI coverage. The required CI checks supply that coverage. ## Risks This changes execution admission. The existing proof requires the stored refusal receipt and no provider work or retained execution resources. Earlier recovery holds, process stop checks, decisions, actor authority, and task ownership still apply. No schema change or migration. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, tool use, and code execution. The host does not expose an exact author model ID or context window. The live acceptance provider was `gpt-6.1-sol` with high reasoning effort. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green — all GitHub Actions checks pass on `c43a84b1bf`; Paperclip Review completed at 5/5 with no findings (neutral check conclusion). - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups — reviewed `c43a84b1bf` with no actionable findings - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
4696ff4990 |
fix: reserve agent In Progress display for active runs (#19453)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Task status must tell operators whether an agent is working or waiting. > - A task can retain its in-progress lifecycle status after its run stops. > - The shared read projection only handled scheduled monitoring waits. > - The UI also ignored other projected statuses for in-progress tasks. > - This pull request requires an active task run for the In Progress display. > - Operators can see waits without changing execution or recovery authority. ## Linked Issues or Issue Description **What happened?** An agent task still showed In Progress after its run ended with an access failure and left a pending connection request. Stale execution locks and saved lifecycle state could show the same false activity. **Expected behavior** An agent task shows In Progress only with a task-linked queued or running run. Otherwise, it shows Blocked for recovery, In Review for a pending response or review, Monitoring for a scheduled check, or Todo for remaining work. Human ownership keeps its selected status. **Steps to reproduce** 1. Create an agent-owned task with an in-progress lifecycle status. 2. End its run and retain a pending human request. 3. Read the task, its summary, and search results. Before this change, they still show In Progress. ## What Changed - Derive visible activity from company-scoped, task-linked runs in the existing shared SQL projection. - Project inactive tasks to existing waiting or todo states. Preserve lifecycle status, locks, admission, and recovery. - Make the shared UI resolver use the projection while preserving explicit and optimistic status changes. - Cover stopped runs, stale locks, human waits, company isolation, monitoring, filters, counts, search, and UI labels. - Document the presentation rule in the product and execution contracts. ## Verification - Red before the server fix: 7 new regression cases failed because they returned in_progress. The 10 existing cases passed. - Green after the server fix: 20 cases passed in issue-display-status.test.ts, using embedded Postgres and real task APIs/search. Seven nearby active-question cases also passed. - Review regression: dismissed requests, historical questions, and returned review work each failed before the follow-up fix. All three now pass using the existing active-interaction rule and pending review state. - Red before the UI fix: 4 label cases failed. Five additional safety cases caught stale projections hiding explicit transitions. - Green after the UI fix: 43 tests passed across task-status-presentation, issue-filters, and issue-detail-subissues. - pnpm -r typecheck passed. All UI token gates and synchronization checks passed. - pnpm build passed. Server typecheck and build passed again after the review fixes. - Full CI passed on |
||
|
DottaandClaude Opus 5.5
|
8c578668b8 |
fix(heartbeat): start new queued runs promptly after restart and name the wait (#19431)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The heartbeat queue starts queued agent turns. Restarts and deploys
happen often.
> - After a restart, new queued turns waited minutes before they
started. The task showed only "Queued", with no reason.
> - The cause is startup order. The queued-run dispatch worker registers
only after many unrelated first passes finish one after another: tool
reviews, delivery workers, the terminal workspace sweep, native restart
recovery, and residual and issue recovery.
> - The server binds its listener before this recovery, and
`/api/health` answers during it. A deployment can route traffic to the
new server for minutes before queued work starts.
> - This pull request starts queued turns that never executed as soon as
plugin workers and adapters are loaded. It also names the wait while the
server starts, recovers, or shuts down.
> - The benefit is that tasks resume quickly after a restart and show a
visible reason while they wait.
## Linked Issues or Issue Description
**What happened?**
On a hosted instance, the server restarted twice for deploys. Both
times, queued runs on idle agents started about 3.5 to 4 minutes after
the restart. Before the restart, the runs also waited about 5 minutes
while the deploy drained the old server. During the wait after the
restart, the execution presentation showed `phase: queued`, `label:
"Queued"`, `cause: null`.
**Expected behavior**
Queued work resumes automatically and quickly after a restart. While it
waits, the task shows a reason.
**Steps to reproduce**
1. Start a task drain (`POST /api/instance/task-drain`). Assign issues
to idle agents. The runs stay queued.
2. Stop the server (SIGTERM) and start it again.
3. Make any startup first pass slow (a large instance does this
naturally). Queued runs start only after every earlier startup pass
finishes.
**Paperclip version or commit**
master at
|
||
|
DottaandClaude Opus 5.5
|
140fcf98ed |
fix(retry): name a declined task Retry and let a new click replace it (#19426)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A task interrupted by a restart or deploy must resume automatically, or offer a Retry that works on first use > - A board Retry saves an intent first. Dispatch then rechecks the requester's authority and saved source before it starts a run > - When that recheck refused the intent, dispatch returned a bare "deferred". The intent stayed saved with no reason, and the task still showed Retry > - Each later click with a new key joined that same intent, so Retry returned 202 and never started a run > - This pull request names the refusal on the intent and lets a new click replace an intent that can never start > - The benefit is that a Retry button that is shown starts one run, or explains why it cannot ## Linked Issues or Issue Description **What happened?** A native task in a managed sandbox lost its runner twice. Saved-work recovery recovered the files. The task showed "Saved work recovered. Retry or send a message to continue." and `canRetry: true`. A Retry click returned 202 and logged `issue.retry_requested`, but no run started. A second click with a new idempotency key returned the same request ID and `state: failed`. No run started. **Expected behavior** If Retry is offered, a click starts exactly one run. If the Retry cannot start, the task names the reason, and a later click is a fresh attempt. A Retry accepted during the pre-deploy drain starts after the restart. **Steps to reproduce** 1. Recover the files of a failed native run so the hold offers Retry. 2. Save a Retry intent whose dispatch-time authority recheck fails (for example, the requester lost access). 3. Dispatch it: no run starts, and the intent has no wait reason. 4. Click Retry again with a new key: the response returns the first request ID, and no run starts. **Paperclip version or commit** Reproduced on `b400914345` (staging served `4d49a31286`; the affected code is the same). Related: #19399 (automatic Retry after a lost runner), #19422. ## What Changed - `heartbeat/queue.ts`: when dispatch declines a task Retry because the requester can no longer change the task, or its saved board message changed, record a named wait (`retry_authority_denied`, `retry_source_changed`) and log it. Before, it returned "deferred" with no record. - `task-retry.ts`: a new click with a new key replaces an unadmitted intent that dispatch declined for its requester's access (`retry_authority_denied`). The old intent becomes `skipped` / `task_retry_replaced`, and activity records `replacedRequestId`. Waits on shared state (cleanup, recovery, an active run, preservation) still coalesce. The restart hook never replaces a user's click. Chat-owned intents keep their own owner. - `task-interruption-presentation.ts`: fixed public messages for the two new wait reasons. `retry_authority_denied` keeps Retry, because a new click is a fresh attempt. `retry_source_changed` is terminal: an edited saved message cannot be replayed, so Retry is hidden and the user sends a new message. - `doc/execution-semantics.md`: document the rule. ## Verification - New regressions in `server/src/services/workspace-recovery.test.ts` use the real recovery, Retry and dispatch services on embedded Postgres. The fixture runs the automatic Retry from a lost runner, then that Retry's own run loses its runner, and recovery saves its files: - `starts exactly one run when Retry is pressed after the automatic Retry's own run lost its runner` (also checks that a replayed dispatch and a second click start no second run). - `does not let a Retry that could not start swallow a later Retry`: red before the fix in two steps. With no fix, the declined intent has no wait reason. With only the wait fix, the second click returned the first request ID. Green after the fix: one run, bound to the second click. - `dispatches a Retry accepted during the pre-deploy drain once the restarted server resumes`: green before and after. The drain does not lose a saved Retry. - `npx vitest run src/services/workspace-recovery.test.ts src/services/task-retry.test.ts src/services/task-interruption-presentation.test.ts src/services/quarantined-task-retry.test.ts src/services/explicit-native-continuation.test.ts`: 892 passed. - Not verified on staging. After deploy, an intent that dispatch refuses shows a named `retryWait` reason. ## Risks - Low risk. Only intents that dispatch declined for their requester's access are replaced. All other coalescing stays. - A replaced intent's late dispatch finds it `skipped` and starts nothing, so a double click still starts at most one run. - The exact gate that held the staging intent is not confirmed from records. If it was a shared-state wait, the new code names it only if it is one of the two reasons above. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, file edits, local test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
7e3caa7bac |
fix(tree-control): resume work a pause gate refused after release (#19421)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - The board can pause an issue tree. A pause stops work. Resume or
release must continue that work once.
> - PR #19394 made release re-dispatch a run that a pause Stop
cancelled.
> - That rule covers only a run that an operator Stop ended and that the
hold snapshot named.
> - A scheduled retry that comes due during a pause is not stopped. The
dispatch gate cancels it with `issue_paused`. The release did not see it
as owed work, so the task stalled with no run.
> - This pull request makes every gate that refuses work for a pause
record the covering hold. Release then re-dispatches that work exactly
once.
> - The benefit is that a paused task with a parked retry or a queued
run continues after release, with no manual wake.
## Linked Issues or Issue Description
Refs #19394
**What happened?**
A child task had a `workspace_busy` scheduled retry. The board paused
the parent tree. The retry came due during the pause. The
scheduled-retry gate cancelled it with `issue_paused` ("Scheduled retry
suppressed because the issue is held by an active subtree pause hold").
The board released the hold. Four minutes later the child was still
`in_progress` with no run, no execution blocker, and no new activity.
**Expected behavior**
Release (or `mode: "resume"`) re-dispatches the refused work exactly
once. No reconciliation and no user message are necessary.
Stranded-issue recovery does not add a second run.
**Steps to reproduce**
1. Give an assigned child task a parked `workspace_busy` scheduled
retry.
2. Pause the parent tree.
3. Let the retry come due while the pause is active. The gate cancels it
with `issue_paused`.
4. Release the pause. No run starts for the child.
**Paperclip version or commit**
master at
|
||
|
Devin FoleyandPaperclip
|
bf53060904 |
fix(recovery): continue saved input after verified workspace copy-back (#19422)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native execution preserves accepted results and task files after a failure. > - A later user message can request a new turn after the old execution stops. > - File recovery can finish by copying files back or by retaining a verified archive. > - The retained-result gate accepted the archive proof but missed a verified copy-back receipt. > - This change accepts that existing receipt for terminal failures and preserves execution authority and prior results. ## Linked Issues or Issue Description **What happened?** A task keeps a user message queued with `finalizing_saved_result` after its old native coordinator reached `terminal_failure`. The accepted result and a successful, exact `workspace_recovery` copy-back receipt are saved. The gate only checks the alternate `workspace_recovery_snapshot` receipt. An older failed result can also keep a newer failed turn's saved input blocked. **Expected behavior** A new user message or deliberate task Retry can continue from verified recovered files after the old execution stops. Preserve the old result and unknown action outcomes. Keep an active finalizer or missing stop evidence blocked. **Steps to reproduce** 1. Retain an accepted result when a native run reaches terminal failure. 2. Complete host copy-back with the exact run, allocation, descriptor and final file digest. 3. Save a new user message or request task Retry. 4. Observe that admission still reports `finalizing_saved_result`. **Paperclip version or commit** Reproduced against `4d49a3128613b282fb344272ed7ca15f98dea2ac`. **Deployment mode** Authenticated native execution with retained workspace recovery. Local regressions use disposable PostgreSQL and the real heartbeat queue. They start no model provider or sandbox. Related work: #19355 covers saved input after Stop. #18267 concerns manual reconciliation of duplicate legacy holds; it excludes workspace and system recovery and does not fix this receipt gate. I searched open PRs and issues before starting this change. ## What Changed - Reuse the existing host-verified repair receipt when an accepted result belongs to a coordinator in `terminal_failure`. - Retain the separate checks for active controllers, environment termination, live executions, questions, approvals and current user authority. - Add message, saved-message and task Retry regressions, plus concurrent queue delivery and invalid-receipt cases. - Document this continuation path. Add no schema, API or background worker. ## Verification - All 483 tests pass across explicit continuation, task Retry, quarantined Retry and finalized recovery migration suites. - The final focused fixture has 17 cases. All pass with the change. With the original production gate restored, all four positive cases fail and all 13 safety cases pass. - Negative cases cover missing, unfinished and mismatched receipts, another task or source, unrecovered files, a live finalizer, an active controller, missing remote stop and active execution. - The real heartbeat path delivers one saved message under concurrent recovery calls. Later recovery calls create no second turn. The old accepted result and recovery budget stay unchanged; unknown outcomes stay unknown. - The first delivery fixture omitted the newer failed source needed by the recovery scanner. I corrected the fixture to preserve that history and repeated the unchanged-code comparison. Earlier failure logs remain in the verification record. - `pnpm -r typecheck`: passed. - `node scripts/check-module-boundaries.mjs` and `git diff --check`: passed. - `pnpm build`: passed. - Full local `pnpm test:run`: running with a 20-minute bound; no full-suite pass is claimed. All hosted CI checks passed. Greptile scored the exact head `d04fb98892fd77853ad03e6b44cd612e805abc13` 5/5 with no findings or review threads. - Hosted delivery of the already saved input still requires a qualified staging build containing this change. ## Risks The new path requires the existing exact repair receipt and a terminal failure. A coordinator still accepting or committing a result keeps the previous hold. Stop, owner and live-run checks still run before admission. The old result is neither applied nor replayed. Receipt verification adds bounded indexed reads only on this retained-result failure path. No migration or provider configuration changes are required. ## Model Used OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) through Codex in T3 Code, with xhigh reasoning. The context-window size is not exposed. Used code editing, tool calls and local test execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used, including the available runtime details - [x] I have checked ROADMAP.md and confirmed this is a scoped bug fix - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in this PR using the bug-report fields - [x] I have not referenced internal or instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal task id - [x] I have run the affected tests locally and they pass - [x] I have added regression tests and updated relevant documentation - [x] I have considered and documented the risks - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open findings - [x] I will address all review findings before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
11bc490cda |
fix(recovery): keep a stopped copy-back's files instead of holding the workspace (#19385)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native run can have its result accepted before the copy-back of its sandbox files finishes. If the copy-back fails, it retries. > - A board Stop during those retries ends the run. It sets the run's finalization to `terminal_failure` / `native_retry_cancelled`, and the result stays accepted. > - After that, nothing exports the files. The finalization worker skips terminal rows, and cancelled-workspace preservation skipped any run with an accepted result. > - The task workspace stays dirty and bound to the stopped run. Every later run on the task waits for the workspace, then fails setup with the unfinished-writer error. > - This pull request lets cancelled-workspace preservation also admit an accepted result whose copy-back a Stop ended. > - The benefit is that stopping a stuck copy-back saves its files and frees the task, like stopping any other run. ## Linked Issues or Issue Description Refs #16700, #17677, #18820, #19304 and #19305 (cancelled-workspace preservation). This pull request adds the accepted-result case to that path and keeps its other checks unchanged. **What happened?** On a hosted instance, a native run's result was accepted, but its copy-back kept failing and retrying. A board operator pressed Stop to unblock the task. The run became `cancelled`, its sandbox was stopped and kept, and its finalization moved to `terminal_failure` / `native_retry_cancelled` with the result still accepted. The next run on the task was cancelled with "Shared execution workspace is busy: run … ended, but its workspace save has not settled". About 24 automatic retries followed over an hour. The last one failed with "The workspace still has an unfinished writer or save operation." After that nothing ran: the task stayed `in_progress` with no run and no recovery action. A Retry or a new comment would hit the same block. The workspace stayed `dirty: true` with the stopped run as its writer, and no `workspaceRestoreRecovery` was ever recorded. The reason: `preserveCancelledNativeWorkspaces` requires `resultId IS NULL`, and `registerCancelledWorkspacePreservationWork` and `preserveNativeFailedWorkspace` both return early when `resultId` is set. **Expected behavior** Stopping a run during its copy-back retries keeps its files, as stopping any other run does. Paperclip's workspace recovery saves them and releases the workspace. Nothing commits or replays the stopped result. **Steps to reproduce** 1. Accept a native result for a run whose copy-back is in `retryable_failure`. 2. Stop the run with the board Stop. The finalization becomes `terminal_failure` / `native_retry_cancelled`. 3. Start another run on the task. It waits for the workspace and then fails setup. No save of the stopped run's files is ever recorded. The new database tests run steps 1 and 2 against PostgreSQL and check that the save is recorded. **Paperclip version or commit** `master` at `9ceaec6f97e`. **Deployment mode** Any deployment that uses remote (sandbox) task workspaces. Seen on a cloud-managed instance. ## What Changed - `server/src/services/native-runtime/native-failed-workspace-recovery.ts`: cancelled-workspace preservation now also admits an accepted result when the finalization is `terminal_failure` with `native_retry_cancelled`. That is the state a Stop leaves after ending copy-back retries. All three entry points use the same rule: - the startup scan `preserveCancelledNativeWorkspaces`; - the worker `registerCancelledWorkspacePreservationWork`; - `preserveNativeFailedWorkspace`, only when it is called with `requireCancelledStop`. Preservation still requires everything it required before: a cancelled run, an acknowledged Stop, no live finalization owner or controller lease, a prepared workspace reference, and the exact retained source. - An accepted result with any other failure code is still skipped; those runs keep their own export repair path. Failed (not cancelled) runs are unchanged. - The accepted result stays recorded and uncommitted. Preservation does not change the task's status or queue a run. - `doc/execution-semantics.md`: describe the stopped copy-back case in the paragraph on writers stopped mid-run. - `native-cancelled-workspace-preservation.test.ts`: move the accepted-result fixture into an `acceptResult` helper, and add a test with three cases. ## Verification - Failing first. The new test "preserves files of an accepted result whose copy-back was stopped" has three cases: worker, startup scan, and a sandbox that the Stop already stopped and kept, through the real `prepareSandboxStopAndRetain` and `settleStopOnlyCleanup`. All three fail on `master`, with no `workspaceRestoreFailure` recorded, and pass with this change. - Each case checks that: - the save obligation and recovery action are recorded for the exact lease; - `inspectWorkspaceRecoverySource` finds one source; - the finalization keeps its `resultId`, `terminal_failure` and `native_retry_cancelled`; - the workspace stays held by the stopped writer until the save publishes; - no run or wake is created, and the task status is unchanged. - The existing "does not admit preservation for accepted-result" case still passes. It uses an accepted result with a different failure code. - `cd server && npx vitest run src/services/native-runtime/native-cancelled-workspace-preservation.test.ts`: 44 passed. - Related suites, 10 files and 854 passed: - `native-failed-workspace-recovery`, `workspace-recovery`, `heartbeat/run-control` - `native-cancellation-request` (unit and integration) - `heartbeat-remote-workspace-busy`, `native-workspace-export-retry`, `native-workspace-reconstruction` - `native-stop-accounting`, `task-workspace-binding` - `cd server && npx tsc --noEmit -p tsconfig.json`: pass. - Not run end to end on a live instance. Diagnosed from that instance's run, finalization, lease, workspace and recovery-action rows. ## Risks - On deploy, the startup scan and the worker will pick up existing cancelled runs in this state and record their saves. That is the intended repair, and it uses the same retention checks as other stopped writers. - A copy-back interrupted by a server restart can leave a `remote_workspace_snapshot` operation marked `running` after its process is gone. This change does not settle such rows. If one blocks the later save, the save follows its existing retry and board hand-off. - No schema migration. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) through Claude Code, with tool use (shell, file edits, test runs, read-only database queries). The runtime does not expose the context window or reasoning configuration. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
90b8190c33 |
fix(native-runtime): detect half-open runner ingress and project Stop for live runs (#19407)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runs in a managed sandbox talk to the server through a provider-ingress WebSocket. The server opens this socket to the runner (`paperclip-runnerd`) through the sandbox preview proxy. > - A staging run ran a 20-minute foreground shell command that printed nothing. After the first second, no run events arrived. The run stayed "Working" for over 35 minutes. No watchdog acted, and the run projection said `canStop: false`. > - The server only detected a lost runner connection from a `close` or `error` event. It never sent pings. A half-open route, for example a proxy that drops the connection without a FIN, never emits `close`. The run then waits forever and the 7-minute reconnect grace never starts. > - A long silent tool call is legitimate. The fix must tell "quiet but alive" apart from "nothing alive". A protocol ping does that: runnerd answers every ping with a pong while its provider waits on the tool. > - This pull request pings authenticated runner connections and terminates the socket after 90 seconds of total silence. The existing reconnect grace and transport-failure path then recover the run. It also makes the execution projection report `canStop: true` for every queued or running run. > - The benefit is that a run with a dead runner connection now ends in a bounded time instead of hanging. API readers also see the Stop that the board already offers. ## Linked Issues or Issue Description **What happened?** A native-runner run (managed sandbox, OpenCode with an OpenRouter model) ran a foreground shell command for about 20 minutes with no output. The last run event came at about T+1s. The command finished on disk, but the run never got another event and stayed `running` / "Working" for over 35 minutes. Its projection had `canStop: false` and `permittedActions: ["inspect_run"]`. A board cancel through the API worked. **Expected behavior** The server detects a dead runner connection and recovers or fails the run in a bounded time. A runner that is alive but quiet stays attached. The projection reports that a live run can be stopped. **Steps to reproduce** 1. Start a native run in a managed sandbox. 2. Make the agent run a long foreground command that prints nothing. 3. Let the preview route drop the WebSocket without a close frame. 4. Before this fix, the run stays "Working" forever. ## What Changed - `server/src/realtime/runner-prp-outbound.ts`: after PRP authentication, `WsJsonWireConnection.monitorLiveness` sends a WebSocket ping every 20 s. Any inbound frame or pong counts as liveness. After 90 s with nothing inbound, it terminates the socket. That emits an ordinary `close`, so the PRP authority and the reconnect loop use the existing 7-minute grace and then fail the transport. If the server's own event loop stalls, the window restarts instead of blaming the runner. The 90 s bound is longer than runnerd's own 60 s connection lease, which already requires runnerd to read the socket. - `server/src/services/execution-projection.ts`: `canStop` is now `true` for every `queued` or `running` run. The old failed-native-retry rule is unchanged. - `server/src/services/execution-projection.ts`: a queued run now shows "Waiting for previous run to stop" (cause `previous_run_settling`) when a predecessor that admission checks (the task execution owner, the retried run, or the agent's latest settled run on the task) still has pending cleanup by `hasPendingRunCleanup`. Before, a turn queued after a Stop showed only "Queued" with no reason while the stopped run's controller still waited on the dead runner. That in-process wait (`claimQueuedRun` settling-owner gate) also made Retry a no-op. The liveness bound above makes that controller settle. Its existing cleanup then calls `startNextQueuedRunForAgent`, which starts the queued turn. - `server/src/services/run-cancellation.ts` and `server/src/modules/run-dispatch/adapters/postgres.ts`: a user-stopped native run whose runner was unreachable can later settle as `failed` (`native_provider_terminal_failed`) when its transport fails closed. The dispatch gate's message deferral required a `cancelled` source and compared the message with `finishedAt`. The user's follow-up message was therefore marked `skipped`, and the task asked for a second message. The new `recordedUserStop` helper honors the recorded user Stop on such a failed source, so the gate saves the message for the deferred-receipt worker. - `packages/shared/src/types/execution-projection.ts`: documents the `canStop` meaning. - `doc/execution-semantics.md`: documents the ingress liveness bound and the new `canStop` meaning. ## Verification - Red then green: `runner-prp-outbound.test.ts` "detects a half-open authenticated ingress and fails the run within a bounded window". On the old code it failed because no ping was sent and the socket was never terminated. It now passes and the active-run `failure` rejects with `runner_ingress_unavailable`. - Safety: "keeps a quiet but live runner attached through a long silent tool call" checks that a runner answering pings stays attached for 20 simulated minutes with no termination and no failure. - Red then green: `execution-projection.test.ts` "offers Stop for every live run, including a long-silent native turn". On the old code it failed with `canStop: false`. - Red then green: `execution-projection-recovery.test.ts` "names the wait when a stopped predecessor still holds the task's runner" (real Postgres). On the old code the successor projected a plain "Queued". It also checks that the wait clears on lease release and ignores another agent's run. - Red then green: `workspace-recovery.test.ts` "continues a message sent after Stop when the stopped run later failed closed". On the old code it failed 3 of 3 runs with the receipt `skipped`. It now passes, and the full file passes twice (392 tests). The test now pins the message after the recorded Stop. Before, the fixture could write both in the same millisecond, which made the sibling variants flaky. - Passed: run-cancellation, explicit-native-continuation, run-dispatch and run-control suites (622 tests). - Passed: `npx vitest run src/realtime/ src/services/execution-projection*.test.ts src/__tests__/heartbeat-run-privacy-routes.test.ts`, the native-runner-execution-driver tests, and the plugin-runner-task tests. - Passed: `pnpm --filter @paperclipai/server typecheck`. - I read runnerd (`runner-core/src/durable/transport.rs`) and confirmed it answers `Ping` with `Pong` in its non-blocking read loop. - Not run: a live managed-sandbox reproduction. Staging and Daytona were not reachable from this environment. `plugin-runner-probe.test.ts` needs a local cargo build of runnerd, and it did not start in this worktree. ## Risks - A false termination is possible if a runner fails to answer pings for 90 s while it is still alive. runnerd already loses its own connection lease after 60 s without reading, so this bound does not add a stricter requirement. A terminated live runner reconnects within the existing grace. - If the preview proxy drops WebSocket control frames, ping alone does not prove liveness. Inbound PRP frames (runnerd's lease renewals, about every 30 s) still count, so a live runner stays attached. - `canStop` now means "the board can Stop now" for live runs. The UI already shows Stop for running and queued runs without reading this hint, so the UI does not change. - Out of scope: the inbound runner WebSocket server (`runner-prp-ws.ts`), and a lost OpenCode SSE event while the runner is alive. A stopped run whose controller hangs for a reason other than a dead transport still holds the queued successor. The projection now names that wait, but this PR does not force-release it, because the controller has no termination evidence. A queued reason for the agent concurrency cap is also not covered. The wait label calls the admission predicate (`hasPendingRunCleanup`) for up to three predecessor runs per queued run. This adds a few reads to projections of queued runs only. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell, file edits) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
a4bb059f59 |
fix(heartbeat): retry transient sandbox provider lease refusals and explain the block (#19412)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A task runs in an execution environment. A managed sandbox
environment gets a lease from a sandbox provider (for example Daytona)
before the agent starts.
> - When the provider answered sandbox create with a temporary error
(HTTP 502), the run failed as the generic `setup_failed`.
> - `setup_failed` is the fail-closed code for unknown setup errors. It
gets no automatic retry, so the task went to `blocked` at once. The task
showed only "Couldn't finish this attempt".
> - A user had to find and press Retry. The Retry then worked, which
shows that the failure was temporary.
> - This pull request classifies temporary provider refusals during
lease acquisition from structured fields. It retries them within the
existing bounded bootstrap budget, and it explains the block when the
retries are used up.
> - The benefit is that a short provider outage no longer stops a task.
When the outage continues, the task shows the cause and a Retry that
works.
## Linked Issues or Issue Description
**What happened?**
A task was assigned to an agent that uses a managed Daytona sandbox
environment. Sandbox creation returned `HTTP 502`, and the cleanup of
the provisional sandbox could not be confirmed. The run failed with
`setup_failed` and the error `Failed to acquire lease for environment
"Paperclip Sandbox" (sandbox): Daytona sandbox creation failed (HTTP
502); cleanup could not be confirmed for paperclip-create-…`. Three
seconds later the task was `blocked`. No automatic retry was scheduled.
The task showed no specific reason. A manual Retry started a new run
that completed normally.
**Expected behavior**
A temporary provider error (HTTP 5xx, 429, timeout) before the agent
starts must retry automatically with bounded backoff. The task must show
why it waits. If the retries are used up, the task must show the cause,
a next action, and a Retry that works on the first click.
**Steps to reproduce**
1. Configure an agent with a sandbox environment.
2. Make the provider return HTTP 502 on sandbox create.
3. Assign a task to the agent.
4. See that the run fails with `setup_failed` and the task goes to
`blocked` with no retry.
**Paperclip version or commit**
`master` at `612da1e8e0`.
## What Changed
- Plugin SDK: an `environmentAcquireLease` error now carries a closed
diagnostic envelope (HTTP status or a closed errno code) across the
worker RPC. It takes the status of the create call only, not of the
compensating cleanup lookup. The existing creation-cleanup ownership
envelope is unchanged. New module: `environment-acquire-error.ts`. The
resume diagnostic helpers are shared.
- Server: new `EnvironmentProviderUnavailableError` and code
`environment_provider_unavailable`. The environment run orchestrator
sets it as the cause of `lease_acquire_failed` only when structured
evidence shows a transient status (408, 425, 429, 500, 502, 503, 504, or
a closed connection errno). It never parses message text. Saved-sandbox
resume and stopped-worker refusals keep their own codes.
- Setup failure (`failRunSetup`): a fresh, pre-dispatch run records
`environment_provider_unavailable` with a `providerFailure` (phase
`startup`, retryability `transient`). The run uses the bounded bootstrap
retry (30 s, then 60 s). An accepted interaction continuation keeps its
own continuation retry, as for `plugin_worker_unavailable`.
- Recovery: the new code is registered with the other bootstrap codes.
It is not sent to the generic continuation retry. It does not create an
execution-reconciliation hold, because no provider work started. On
exhaustion the stranded notice has the title "Sandbox provider was
temporarily unavailable" and a specific next action.
- UI: `ProviderFailureNotice` labels the `sandbox` provider as "Sandbox
provider". The task notice shows the provider failure, "Paperclip will
retry automatically" while a retry is scheduled, and Retry after
exhaustion.
- Docs: `doc/execution-semantics.md` describes the policy and the
sandbox ownership rule.
## Verification
Red before the fix, green after the fix:
- `server/src/__tests__/heartbeat-process-recovery.test.ts` › "transient
sandbox provider refusal during lease acquisition":
- "retries automatically and runs exactly one successful successor
without user action": the real heartbeat. The lease runtime returns a
Daytona 502 once, then delegates. Before the fix: one `failed` run and
no successor. After the fix: one `scheduled_retry` successor with no
hold and a `retry_scheduled` presentation. After promotion, exactly one
successor succeeds.
- "explains the block with a working Retry once automatic retries are
used up": three refusals. The task is `blocked` with a "Sandbox provider
was temporarily unavailable" notice. The presentation is `{ state:
"failed", canRetry: true, providerFailure: { httpStatus: 502 } }`, and
no execution-reconciliation hold exists. One `requestTaskRetry` is
accepted, and its run succeeds.
- `server/src/services/heartbeat/run-completion.test.ts` › "routes a …
sandbox provider lease refusal through bounded bootstrap recovery": the
real orchestrator wrapping. Only the fresh, structured 502 is retried. A
message-only 502, a 400, a run after provider dispatch, and a retained
native run stay `setup_failed`.
- `packages/plugins/sdk/tests/environment-acquire-error.test.ts`: the
real worker RPC. The test checks the create status (not the 404 cleanup
lookup), the errno, no provider text, the unchanged cleanup envelope,
and that forged envelopes are rejected.
Commands:
```sh
pnpm --filter @paperclipai/plugin-sdk exec vitest run tests/environment-acquire-error.test.ts tests/environment-creation-cleanup.test.ts tests/environment-resume-error.test.ts
cd server && npx vitest run src/__tests__/heartbeat-process-recovery.test.ts src/services/recovery/ src/services/heartbeat/run-completion.test.ts src/services/heartbeat/retries.test.ts src/services/legacy-execution-recovery.test.ts src/services/task-interruption-presentation.test.ts src/services/task-retry.test.ts src/__tests__/environment-runtime.test.ts src/__tests__/heartbeat-retry-scheduling.test.ts
cd packages/plugins/sandbox-providers/daytona && npx vitest run
pnpm --filter @paperclipai/plugin-sdk typecheck && pnpm --filter @paperclipai/server typecheck
```
I did not run a live Daytona 502 on staging. Re-check on staging after
deploy (see Risks).
## Risks
- Sandbox leaks or double creation: each retry is a new run with a new
durable creation record and a new provider name
(`paperclip-create-<operationId>`). The failed attempt stays owned by
its own record, and the existing creation reconciler settles it. A retry
never reuses or abandons it. While a provisional sandbox waits for
reconciliation, up to two more sandboxes can be created for the same
task. This is the same as a manual Retry today.
- Plugins built with an older SDK do not send the envelope. Their
refusals stay `setup_failed`, which is the current behavior.
- An in-process error with `status`/`statusCode` 500–504 or a connection
errno during lease acquisition is now retried twice. The budget is
bounded and shared with the other bootstrap codes.
- An RPC timeout on the host side is not classified by this change.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code as an agent with tool use (shell, file edits, test runs).
Extended reasoning was on.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandClaude Opus 5.5
|
9668e73b4b |
fix(native): continue a turn automatically when reattachment loses its runner (#19399)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs in managed sandboxes recover through reattachment: after a server restart, and for a same-run retry when a turn ends without a semantic result > - Reattachment fails closed when the original runner or sandbox is gone (`native_remote_recovery_lease_mismatch`, `runner_remote_process_identity_unavailable`) > - Saved-work recovery then saves the files, but the task stays blocked with "Saved work recovered. Send a message to continue." The recovered hold offers no Retry > - A restart, deploy or maintenance window must not need a user message to continue work > - This pull request requests the task Retry automatically, once, when it is safe, and offers Retry on the recovered hold when it is not > - The benefit is that interrupted work continues after infrastructure events, and a user who must act gets a Retry that works on first use ## Linked Issues or Issue Description **What happened?** A native task in a managed sandbox was interrupted. In one case a server restart claimed a queued user-comment turn for reattachment. In another case a turn completed without a semantic result, and the same-run retry claimed the run for reattachment with no restart. In both cases the original runner or sandbox lease was gone. The run failed with `native_remote_recovery_lease_mismatch` or `runner_remote_process_identity_unavailable`. Recovery saved the work. The task stayed `blocked` with "Saved work recovered. Send a message to continue." The execution blocker had `canRetry: false`. A manual `POST /api/issues/:id/retry` started a run that succeeded. **Expected behavior** After an infrastructure interruption, the task continues automatically when the interrupted turn's outcome is known. Otherwise the task offers a Retry that works on first use. **Steps to reproduce** 1. Start a native task in a managed sandbox from a user comment. 2. Lose the sandbox lease or the remote runner process (for example, restart the server during a maintenance window). 3. Let reattachment fail and saved-work recovery complete. 4. See that the task waits for a new message and offers no Retry on the execution blocker. **Root cause** Reattachment (`reattach_remote_runner`) correctly refuses to replace a missing runner or allocation. Nothing then continued the user's request. The automatic continuation rule only allows a new user turn after reconciliation. `getExecutionBlocker` hid Retry whenever a workspace restore hold existed, also after the files were verifiably recovered. The same-run retry path also re-enters the restart-recovery owner, so its event said "after server restart" with no restart. ## What Changed - New `server/src/services/heartbeat/restart-interrupted-turn.ts`: after a reattachment claim executes, if the run failed because its runner or allocation was lost, request the same task Retry that `POST /issues/:id/retry` creates. It runs once per run (idempotency key `restart-interrupted-turn:<runId>`), under the authority of the user who sent the message, assigned the task, or clicked Retry. It requires that the run recorded no provider turn or that its last provider turn is settled. System wakes, in-flight turns, and a Retry that this hook already requested do not continue automatically. - `heartbeat/recovery.ts`: call the hook after each reattachment claim. Label same-run retry reattachment as "Reattaching to the existing remote runner to resume this run", and add `trigger: "same_run_retry"` to the event payload. - `execution-blocker.ts`: a failed run whose saved work is verifiably recovered now offers Retry (`canRetry: true`) with "Saved work recovered. Retry or send a message to continue." An intentional Stop keeps "send a message" only. - `task-retry.ts`: optional `initiatedBy: "restart_recovery"` records the activity as actor `system`/`restart-recovery` with `onBehalfOfUserId`. - `doc/execution-semantics.md`: document the rule. ## Verification - New regressions in `server/src/services/workspace-recovery.test.ts` use the real recovery, Retry and dispatch services on embedded Postgres: - `continues a user turn once ... after $title and recovery saved the work` (3 cases: restart lease loss before the turn, same-run retry with the runner gone after a settled turn, same-run retry with the lease gone after a settled turn). Each asserts one queued successor bound to the Retry and the user's authority after recovery, idempotent hook calls, system activity attribution, and no second automatic attempt for the automatic Retry's own failure. - `does not continue automatically when $title, and offers a Retry that works on first use` (in-flight turn, system wake, unrelated error). Each asserts no successor, `canRetry: true` on the blocker and the interruption presentation, and one successor after one Retry click. - Red before the fix: all 6 tests fail (no successor, blocker `canRetry: false`). Green after the fix. - `npx vitest run src/services/workspace-recovery.test.ts src/services/task-retry.test.ts src/services/task-interruption-presentation.test.ts src/services/execution-blocker-read.test.ts src/services/heartbeat/recovery.test.ts src/services/explicit-native-continuation.test.ts src/services/quarantined-task-retry.test.ts src/__tests__/workspace-restore-repair-route.test.ts src/services/native-runtime/native-failed-workspace-recovery.test.ts`: 1098 passed. - Not verified on staging. After the next deploy, check a task interrupted by a restart: it should continue without a message, or show Retry. ## Risks - The automatic Retry uses the original user's authority. Admission rechecks that user's current permission, stop proof, saved-work recovery, approvals, pauses, budget and ownership. A refused request leaves the visible Retry. - A settled turn that ended without a result is continued with a fresh Retry turn from saved work. A turn whose outcome is unknown (in flight) is not. - The hook runs after the claim's execution returns. If the server stops between the failure and the request, the task shows Retry instead. - Not in scope: same-run disposition recovery still cannot start a new runner in the existing sandbox after the original remote runner exits. This change continues the task with one fresh Retry turn instead. If that Retry turn also ends without a result and loses its runner, the task shows Retry. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, local test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Related: #19355 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
5c185db77c |
fix(tree-control): resume work a subtree pause interrupted (#19394)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Task tree control lets the board pause a task tree and then resume
it.
> - A pause stops the runs that are active in the tree. The release must
then continue the work that was pending.
> - A child task whose run was active at pause time did not start again
after release. The child became blocked and asked for a user message.
> - A pause is an operator hold, not a failure. It must not strand work,
and it must not require reconciliation for a run that never started.
> - This pull request re-dispatches interrupted work one time on
release, and it settles a legacy Stop that never entered the adapter.
> - The benefit is that pause and resume keep a task tree moving without
manual recovery, and real safety gates stay in force.
## Linked Issues or Issue Description
**What happened?**
A parent task delegated two children. One child's run was active (in
host workspace preparation, before the adapter started) when the board
paused the parent's tree. The pause cancelled that run. After the board
released the hold, no new run started for that child. The child became
`blocked` with the execution blocker cause
`legacy_execution_requires_reconciliation`. Later the next action became
"Send a message to continue." A sibling child that had no active run at
pause time resumed normally.
**Expected behavior**
A run that a subtree pause cancels is an operator hold. On release, the
pending work starts again one time, automatically. It does not go
through execution reconciliation when the run never entered its adapter.
It does not wait for a user message.
**Steps to reproduce**
1. Create a parent task and a child task that is assigned to an agent.
2. Wake the child. While its run is in host preparation (or queued),
call `POST /api/issues/:parentId/tree-holds` with `{ "mode": "pause" }`.
3. Call `POST /api/issues/:parentId/tree-holds/:holdId/release` without
`wakeAgents`.
4. Before this change, no new run starts, and `getExecutionBlocker`
reports `legacy_execution_requires_reconciliation` for the child.
**Root cause**
- (a) The pause records each interrupted run in the hold member snapshot
(`activeRunId`). Nothing used that record on release.
- (b) Release woke tasks only when `metadata.wakeAgents === true`.
Without it, release relied on stranded-issue recovery. That recovery
stands down when the latest run was cancelled by an operator
(`isOperatorCancelledRun`). So the interrupted child had no owner.
- (c) The pause Stop of a legacy run in preparation is written directly
as `cancelled`. Legacy terminalization classifies a cancelled run with
no bootstrap evidence as an unknown outcome. It records
`legacy_execution_requires_reconciliation`, although the gated dispatch
never entered the adapter.
- Native runs: the verified native Stop does not create the
reconciliation hold, but gap (b) applies to native runs too. The release
change is runtime-independent.
## What Changed
- `issueTreeControlService.findOwedPauseResume` decides if a task is
owed one resume. The task's latest run must be an operator Stop that a
pause hold member snapshot names, recorded while that hold was active.
The task must still be executable by the same agent, and no pause may
still cover it. The obligation belongs to the run, so overlapping pauses
keep it until the last covering release.
- New `services/issue-tree-resume.ts` re-dispatches owed work with wake
reason `issue_tree_resumed` (`holdId` and `interruptedRunId` in the
context). A compare-and-set claim (`treeHoldResume`) on the interrupted
run makes the resume happen at most once. A failed wake returns the
claim.
- Pause release and `mode: "resume"` call it for every member of the
released hold, also without `wakeAgents`. When `wakeAgents` is true, the
existing loop wakes every executable member.
- A task that still has an execution blocker is not replayed. The
release writes `issue.tree_resume_wake_skipped`, the blocker stays
visible, and the obligation stays unclaimed.
- `releaseHold` updates only an `active` hold. A repeated release gets
`409`.
- `settleCancelledLegacyPreparation`: when the executor that owned a
cancelled legacy run proves that dispatch never entered the adapter, it
records bootstrap evidence and resolves that run's
`legacy-execution:<runId>` hold as `false_positive`. It also corrects an
automatic no-replay disposition and restores a `blocked` status only
when that disposition made the change. Then it re-dispatches owed work,
for a release that happened while the Stop was unwinding. It requires a
requested Stop with no existing recovery receipt, and no process,
session, native finalization, unreleased lease, or execution-start
event.
- `settleUnrecoverableExecutions` waits while a legacy executor still
holds its controller lease, and records a `legacyCancellationBlock`
receipt when it blocks a task for a cancelled legacy run.
- Docs: `doc/execution-semantics.md` and `doc/composer-stop.md` describe
the resume rule.
## Verification
- New regression
`server/src/__tests__/issue-tree-pause-resume-run.test.ts` uses embedded
Postgres, the real heartbeat, and the real tree-control routes:
- A run paused during preparation: nothing runs during the hold, there
is no blocker, and release creates exactly one `issue_tree_resumed` run
that succeeds. A second release returns `409` and creates no run.
- Release before the stopped preparation finishes unwinding (with the
periodic disposition run in between): exactly one resumed run after the
executor settles.
- An automatic no-replay disposition that ran first: the blocker clears,
the status returns to `in_progress`, and release resumes once.
- Overlapping child and parent pauses: no resume while the parent pause
remains, then exactly one resume after it is released.
- A queued run cancelled by the pause: release creates exactly one
resumed run.
- A task reassigned to a user during the hold: no resume.
- A mid-turn legacy Stop with unknown outcome: no replay, the
reconciliation blocker remains, and `issue.tree_resume_wake_skipped` is
logged.
- Red before each fix: the first 4 cases failed on master; the 3 review
cases failed on the first PR commit. All 7 pass now.
- Also passed locally: `issue-tree-control-routes`,
`issue-tree-control-service`, `heartbeat-workspace-busy`,
`heartbeat-undispatched-execution-wait`, `execution-control-work`,
`legacy-workspace-restore-recovery`, `legacy-continuation-authority`,
`task-retry`, `legacy-execution-recovery`, `agent-conversations`,
`heartbeat-retry-scheduling`, `satisfied-connection-intents`.
- Not verified here: a live staging pause/resume. Re-check on staging
after deploy.
## Risks
- Behavior change: release now starts the work that the pause
interrupted, also without `wakeAgents`. Before, that work stayed idle.
Other idle tasks still wake only with `wakeAgents`.
- The settlement trusts the executor's adapter-entry flag. The flag is
set only inside the atomic dispatch gate, which refuses a cancelled run.
The settlement also requires no process, session, lease, or execution
event.
- A native run whose Stop has not finalized at release time is not a
target. Stranded recovery and explicit wake still apply.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edit, test execution).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandPaperclip
|
7c0922a7b6 |
fix(native): preserve provider backups during slow workspace saves (#19359)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Remote native sessions save provider state so another web server can continue the same conversation. > - A provider can suspend while its accepted result is still saving a large workspace. > - The run has released its live lease at that point. A new backup-owner lookup rejects this uncommitted state. > - This pull request captures the real owner during backend creation and retains it with the current session authority. > - The existing database checks still reject a changed boot, generation, result, or successor before inspection and during publication. > - The benefit is that a slow workspace save no longer prevents the provider backup. ## Linked Issues or Issue Description **What happened?** A remote session can fail to save its provider backup with `native_harness_storage_owner_missing`. The provider has suspended, but its accepted result is still in `workspace_finalizing`. The close callback tries to acquire a new owner after the run released its lease. A later allocation cannot use a backup that was never published. **Expected behavior** Save the suspended provider with the authority acquired during execution. Recheck that authority before provider inspection and before and after upload. Keep result finalization and provider-work permissions unchanged. **Steps to reproduce** 1. Start a remote native session with durable provider storage. 2. Accept its result and release the live lease while its workspace save remains pending. 3. Close the provider transport. The old owner lookup rejects the save before provider inspection. **Paperclip version or commit** Reproduced on master `bf10bc695abaef6b20e85b287f456b5ce2edb266`. **Deployment mode** A web server with object storage and remote managed Runner sessions. Related: #19225 retains failed-turn ownership through close. #18436 restores an external controller checkpoint before a changed warm process is replaced. This change fixes late backup-owner acquisition while an accepted result saves its workspace. ## What Changed - Capture the provider backup owner when the remote backend still owns its execution lease. - Bind that owner to the current session authority. An older close callback selects the current warm attachment's owner. Revalidate the captured fence before inspecting or skipping provider state. - Add a regression for the close callback during workspace finalization. - Restore encrypted provider files with a captured owner while workspace finalization remains pending. Reject boot, generation, result, and successor changes during upload. - Document the ownership contract. ## Verification - Observed the callback regression fail on the old code with `native_harness_storage_owner_missing`. The same test passes with this fix. - `cd server && node ../node_modules/vitest/vitest.mjs run --config vitest.config.ts src/services/native-runtime/native-harness-backup-storage.integration.test.ts`: 59 passed, using real PostgreSQL and encrypted object write/readback. - `node scripts/check-module-boundaries.mjs`: passed. - Full workspace typecheck: passed with the complete production fix. - Native execution: 1,151 passed. Managed warm lifecycle: 9 passed. These tests require a lost-owner error during shutdown, including before an incomplete provider checkpoint is skipped. The final change to the lifecycle test also requires rejection after lease renewal is lost. - Full local `pnpm build`: passed on the final head. - The full local `pnpm test:run` stopped in chat shard 6 on five existing deadline assertions. Earlier chat shards passed; the PR CI chat shards passed. This is not a full local-suite pass. - The first CI server shard 4 attempt failed the unchanged legacy environment-inheritance fixture at its five-second completion wait. The same failure reproduced locally on the repair head and on an earlier checkout without this repair. One unchanged CI retry passed. No fixture assertion or timeout change is shipped. - Latest-head CI has passed. Greptile reviewed the final head at 5/5 with no open threads. The separate Paperclip Review is paused. This PR does not claim staging acceptance. ## Risks The backup owner now lives with its session authority. A wrong authority selection must fail rather than publish provider state. Database publication still checks the original company, task, run, boot, generation, and accepted result before and after upload. Tests change the owner during upload and verify that the selected backup stays intact. No schema, data migration, or new provider-work permission changes. ## Model Used OpenAI Codex, GPT-6 model family, with reasoning and code execution tools. The host does not expose the exact runtime model ID or context-window limit. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
002455ea06 |
fix(issues): finish a reassignment whose native stop settles after the request (#19354)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A user can stop or interrupt a running native task and then send a message to give new direction > - When the stopped run left a workspace hold, the dispatch gate withholds the follow-up run until recovery saves the work > - The gate must save the user's message on its receipt so that the deferred-receipt worker continues it after recovery > - Two conditions made the gate skip the message instead of saving it, so the task stayed in progress with "Saved work recovered. Send a message to continue." although the user already sent one > - This pull request saves the message in both conditions > - The benefit is that the message the user already sent continues once, automatically, after recovery ## Linked Issues or Issue Description **What happened?** After a user stopped or interrupted a native task with an isolated sandbox workspace and sent a message, the follow-up run was cancelled with `execution_reconciliation_required`. Its error said "Saved work recovered. Send a message to continue." Nothing re-woke the task. In the interrupt case, a periodic sweep started a run about four minutes later. In the case where the follow-up first waited for the stopped run's workspace save (`workspace_busy`), each later retry parked on the same resolved recovery action, and the task did not continue. **Expected behavior** When the recovery that the follow-up waited on resolves, the message that the user already sent continues once, automatically. The product does not ask for a second message. **Steps to reproduce** 1. Run a native task in a managed sandbox with an isolated workspace. 2. Stop the run (or send a comment with interrupt), then send a message while the stop and workspace save are still settling. 3. Wait for the workspace recovery to resolve. 4. See that the follow-up run is cancelled with `execution_reconciliation_required` and that no run continues the message. **Paperclip version or commit** `master` before this change. **Root cause** `deferUnstartedUserComment` (run dispatch) refused to save the withheld message in two cases: 1. It compared the message time with the source run's `finishedAt`. A native run finishes only after its provider stop and workspace save settle, which takes seconds. A message written after the Stop request but before that settlement was treated as input written before the Stop. The receipt was marked `skipped`. 2. It required the run's own receipt to be user-owned. A workspace-busy retry or an admission retry carries the user's message under a system-owned receipt, so the message was dropped with the retry. ## What Changed - Add `userStopBoundaryAt(run)` in `run-cancellation.ts`. For a run cancelled by a user Stop or interrupt, it returns the recorded Stop request time (`cancellation.recordedAt`) when that is earlier than `finishedAt`. For all other runs it returns `finishedAt`. - Use this boundary in the dispatch gate's message deferral, in native explicit-continuation admission, and in the remote-Stop continuation check, so the three places agree on "message written after the Stop". - When the withheld run is a system-owned retry, walk a bounded chain (max 8) of retries that did not reach the provider (`workspace_busy` with `providerWorkStarted: false`, or a verified pre-dispatch `execution_reconciliation_required`) to the originating user receipt. Save the message on that receipt with the user's original wake context. Mark the retry's system receipt `skipped`. - Update the "input written before the Stop" test so that the Stop request, and not only the settlement, comes after the message. ## Verification - New regressions in `server/src/services/workspace-recovery.test.ts` (`continues a message ... without a second message, after recovery already resolved`): a direct follow-up and a workspace-busy retry. Each runs recovery to completion first, then lets the gate withhold the follow-up, then runs only the deferred-receipt worker. Each asserts that exactly one successor run is queued with the user's message and the user's authority, and that no provider work starts. - Red before the fix: both tests fail because the receipt is `skipped` instead of `deferred_issue_execution`. Green after the fix. - `npx vitest run src/services/workspace-recovery.test.ts` (384 passed). - `npx vitest run src/services/explicit-native-continuation.test.ts src/modules/run-dispatch src/services/execution-continuation.test.ts src/services/run-cancellation.test.ts` (581 passed). - 19 more heartbeat, retry, queued-comment, and native recovery suites (1272 passed). One failure, `heartbeat-process-recovery > redacts opaque environment-bound credentials from Sentry diagnostics: result`, also fails on unchanged `master` and is not related. ## Risks - A message written after the Stop request but before the stopped run settled now continues. The stopping run cannot consume new input after its Stop request, and admission still removes comments that a run already delivered (`undeliveredLegacyUserCommentIds`). All admission gates (holds, stop proofs, approvals, budget) still apply. - Only user Stops and interrupts move the boundary. Other cancellations keep `finishedAt`. - The retry walk is bounded, stays in one company, agent, and issue, and accepts only retries without provider work. It revives a user receipt only from `cancelled` or `skipped`, so a second retry cannot save the same receipt twice. - Not in scope: a run that fails during native preparation with no user message (for example `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate; retirement_refused=unexpected_root_entry` on a system wake). After its recovery resolves, the task still waits for a user message. That is a separate root cause in the native session executor's prior-owner retirement proof. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, local test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
789a71bc17 |
fix(runner): retry quarantined tasks in their saved workspace (#19340)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native sessions retain execution ownership and action history across turns. > - Task Retry must prove that old execution has stopped before it starts another turn. > - An attempt can inherit its predecessor's identity and fail before it launches a provider. > - The cleanup proof searched under that failed attempt and compared unrelated event sequences. > - This pull request verifies the retained owner and starts one fresh authorized turn. > - Saved results and uncertain action outcomes remain intact. ## Linked Issues or Issue Description **What happened?** Task Retry accepted repeated clicks but kept the canonical request deferred with `executionWait.reason: local_cleanup`. It created no successor, although the retained local worker and provider had stopped. **Expected behavior:** After exact cleanup proof, admit one fresh turn. Preserve the old journal and result. Keep unresolved action outcomes unknown. **Steps to reproduce:** 1. Finish a local native turn with retained cleanup state and a saved result. 2. Let the next attempt inherit its identity and fail with `native_session_cleanup_quarantined` before any launch. 3. Upgrade across a tool-contract change, then click task Retry. 4. The old code requires the failed attempt's stop receipt and journal identity. It also requires the current tool fingerprint and equal sequence numbers across raw and normalized events. Those requirements cannot match the retained owner. **Paperclip version or commit:** Reproduced on `31bf6e4482`. **Deployment mode / installation method:** Authenticated, self-hosted source checkout. **Agent adapter(s) involved:** Native `paperclip_runner`, local Codex app-server. The shared physical-stop proof also covers ACPX. **Database mode:** Embedded PostgreSQL 18.1. **Access context:** Board task Retry. **Node.js version / operating system:** Incident: Node 24.20.0, Linux ARM64. Local verification: macOS ARM64. **Relevant logs or output:** Accepted Retry stays in `deferred_issue_execution`; no new run or provider launch occurs. **Relevant config:** Per-turn local native execution. The old tool contract differs from the current contract. **Additional context:** Related: #19311 prevents the oversized tool result that can lead to retained cleanup. #18217 covers broader cleanup isolation and retained-result maintenance. #19134 admits stopped Runner sources under the current run lease. This patch fixes explicit continuation admission on current master. ## What Changed - Resolve the retained journal's owner under the same company, task, agent, workspace, native session, and runner instance. - Require terminal ownership, settled controllers, released local leases, and exact inherited process birth. Refuse any launch evidence on the failed attempt. - Accept a current host's verified process absence from its exact launch receipt when an older server lacks a stop receipt. A later launch invalidates that observation. - Validate normalized receipts against their own IDs, sequences, and digests. Match ordered provider identities to the independent raw journal. - Keep tool compatibility at the session reuse boundary. Explicit continuation always uses a fresh session. - Bind the stop proof to its exact successor and canonical Retry receipt. Reuse the proven local workspace through normal preparation. Refuse changed bindings, closed or remote workspaces, pending operations, and other live leases. Projectless run-ID placeholders keep the existing fresh-workspace fallback. - Retain journals, results, files, and uncertain action outcomes. Add regression and opt-in live acceptance coverage. - Make the split-handoff browser test wait for both child-completion notices and all run settlement before teardown. Assert the final task set stays unchanged. ## Verification - Red: canonical Retry on the original code accepts one saved intent but creates zero successors and records `local_cleanup`. - Red: distinct raw and normalized event sequences reproduce that wait after owner resolution alone. After admission was repaired, live acceptance then exposed the empty-attempt workspace problem. A PostgreSQL regression also reproduced that wrong workspace selection. - Green at `6684df5e82`: **479 targeted cases**, including 26 new PostgreSQL cases for duplicate clicks, exact process and workspace ownership, controller/lease fences, changed state, and independent receipt integrity. The complete 1,150-test native executor suite passed earlier. - `pnpm -r typecheck` and `pnpm build` pass again at final head `432a52bea8`, including its build stamp. Token gates pass. Master now contains the unrelated directory-fixture type correction found during the first typecheck. - Real authenticated HTTP Retry with Codex `gpt-6.1-sol`, high effort, and per-turn execution passes again at final head `432a52bea8` (93.3 seconds). It preserves the first file line, appends the second once, starts a fresh session in the existing workspace, keeps the retained journal unchanged, commits finalization, and clears task locks. - The strengthened split-handoff browser scenario passes **three consecutive local runs**. Its original CI teardown and one same-head rerun failed when cleanup cancelled an active legacy completion-notification run and hit a PostgreSQL row-lock timeout. The scenario now waits for the complete handoff before cleanup. [Initial failure](https://github.com/paperclipai/paperclip/actions/runs/42106466518/job/127523693634), [same-head rerun](https://github.com/paperclipai/paperclip/actions/runs/42106466518/job/127531698321). These failures remain part of the verification record. - The local full `pnpm test:run` passed all six chat integration shards. Its obsolete pre-rebase attempt was stopped during the general server phase; no complete local-suite pass is claimed. Full CI, including both required gates, passed at `6684df5e82`. At final head `432a52bea8`, all initial CI groups passed except the unchanged status-card deadline test, whose post-wakeup clock assertion missed its bound by 10 ms. That exact case passed three unchanged local repetitions (16.8/27.0/18.3 seconds); the single isolated shard rerun passed all **2,218 tests** (123 files), with no source changes. [Initial timing failure](https://github.com/paperclipai/paperclip/actions/runs/42114473652/job/127550722187). - Greptile reviewed `6684df5e82` at **5/5**, with no actionable findings. A second reviewer identified the supported projectless placeholder case. A new real PostgreSQL regression reproduced its workspace-binding failure before the one-line fallback correction; all **45 focused cases** now pass, including default environment selection and changed binding/cwd refusal. Greptile reviewed final head `432a52bea8` at **5/5**, with no blocking findings. The second reviewer’s thread is resolved. - The patched Linux build verified the actual retained worker/provider ownership and process absence in a transaction that was rolled back. No retained state was retired or provider started during that pre-deployment check. ## Risks - This changes an execution admission proof. Incorrect ownership could permit a second executor. The proof retains process, controller, lease, scope, journal, and receipt fences, and duplicate admission is tested through the real queue. - Missing or conflicting evidence remains deferred. Windows, remote execution, automatic replacement, and session reuse retain their existing policies. - No database migration or UI change. The original affected task still needs deployment of the fix before recovery can be claimed. ## Model Used - OpenAI GPT-6 through Codex, with reasoning, repository tools, code execution, SSH diagnostics, and live provider testing. This session does not expose the exact backend model build or context-window size. The live test agent uses OpenAI `gpt-6.1-sol` at high effort. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
15249a689d |
fix(server): preserve sessions across cosmetic sandbox updates (#19349)
## Thinking Path > - Paperclip manages agent work and saves provider sessions between turns. > - Session and workspace compatibility include the selected environment revision. > - The managed sandbox reconciler also updates names, descriptions, and stock metadata. > - Those cosmetic updates advanced the revision even when the execution configuration stayed the same. > - A later turn therefore discarded an otherwise compatible saved session. > - This pull request keeps that revision stable for cosmetic reconciliation and retains resets for real configuration changes. ## Linked Issues or Issue Description Related: #18943 and #18552. **What happened?** A managed sandbox rename or description update made a later task turn start a fresh provider session. The driver, provider, snapshot, and agent configuration could all remain unchanged. Both session and workspace freshness use the environment's `updatedAt` as its configuration revision. **Expected behavior** Cosmetic managed reconciliation preserves compatible task sessions. Actual sandbox configuration or availability changes retain the normal replacement checks. **Steps to reproduce** 1. Create a managed sandbox and save a task-session fingerprint for it. 2. Reconcile the same sandbox configuration with a new name, description, or managed metadata. 3. Evaluate the saved session against that environment. Before this fix, the decision resets the session and reports the environment category. **Paperclip version or commit** Reproduced on master `d3713d827f70ddba63a323e57c562384dbd648ba`. **Deployment mode** Managed sandbox server. The regression uses an isolated PostgreSQL database and the actual reconciler and session-freshness resolver. ## What Changed - Keep the existing revision for an active managed sandbox when its execution configuration stays equal. - Continue advancing the revision when configuration or availability changes. - Retain activity records and managed-stock binding updates for cosmetic reconciliation. - Add a regression that checks session reuse, audit history, and rejection after a real snapshot change. - Document the freshness contract. ## Verification - The new PostgreSQL regression failed before the production change: it reported `reset: true` and changed category `environment`. - All 53 environment-service cases pass after the fix. - All 190 session-freshness and assembled heartbeat session-preparation cases pass. - Full local typecheck passes. The full local Vitest run remains active, with its local build queued afterward. Initial typecheck setup failures were retained and resolved by repairing generated SDK links; no source workaround was added. - [Full current-head CI](https://github.com/paperclipai/paperclip/actions/runs/42114984839) passes, including the sharded test suites and build: 55 successful checks, 3 skipped, and 1 neutral. - Greptile and Ponytail both reviewed this commit at 5/5 with no findings. No review threads remain unresolved. - Deployed verification remains pending normal delivery. No pinned release or manual rollout is required. ## Risks - Cosmetic managed reconciliation keeps `environments.updatedAt` unchanged, so it does not reorder that environment by this timestamp. Activity records and managed-resource bindings still record the update. - The comparison uses canonical configuration hashes, so JSON property order cannot hide a real change or create false drift. - Existing fingerprint formats and saved session records remain compatible. Real configuration and availability changes still advance the revision. ## Model Used OpenAI GPT-6 through Codex desktop, with reasoning and terminal/browser tools. The host does not expose a more specific deployment suffix or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
b929ad43de |
fix: keep dependency waits neutral and explain GitHub access failures (#19315)
## Thinking Path > - Paperclip manages agent work and shows users what can move it forward. > - Task status and tool approval cards describe different parts of that work. > - A queued dependency wait has no provider attempt to retry. > - GitHub can reject an approved action while other connection tools still work. > - These cases showed a false Retry banner and an unhelpful provider error. > - This pull request keeps dependency waits neutral and gives GitHub access failures a direct repair path. ## Linked Issues or Issue Description Related: #19268 preserves pending Retry preparation and current failure causes. **What happened?** A queued wake cancelled by the dependency gate appeared as a failed attempt with Retry. A GitHub MCP permission denial appeared as raw HTTP error text after an approved repository action. **Expected behavior** Keep the task's dependency state. Show GitHub access failures with a clear explanation and a link to the selected connection's Permissions page. Preserve actual failures, provider details, and approval gates. **Steps to reproduce** 1. Queue a wake, then leave its task dependent on unfinished work before dispatch. 2. Open the task after the dependency gate cancels that wake. 3. Separately approve a GitHub action whose MCP result contains `403 Resource not accessible by integration`. 4. Reopen the saved tool approval. ## What Changed - Recognize both queues' unstarted dependency-wait receipts in cancellation provenance and task projections. No dispatch or recovery gate changes. - Classify GitHub's specific permission error after MCP result normalization. Keep result validation, the raw diagnostic, and working tools available. - Apply the same display code to both historical GitHub error codes. Identify the provider from company-scoped connection metadata, including renamed apps, without rewriting stored decisions. - Show a token-based inline access card with **Manage GitHub access** and collapsed **Error details**. - Give agents trusted recovery steps after an approved GitHub action fails. Keep provider text in the untrusted result and prevent automatic replay. - Add regressions, a production-component Storybook case, and execution/connector documentation. ## Verification - Red before the fix: cancellation provenance, dependency-wait projection, the real queue cancellation, approved MCP failure classification, saved interaction projection, and the error card assertions failed at the expected behavior. - Green focused tests: queue cancellation, one ready-task executor, approval-before-dispatch, one failed GitHub request, connection reuse, retained provider diagnostics, saved interaction reads, task list/detail projection and redaction. - UI: 106 component tests passed. Token gates and Storybook catalog checks passed. - `node scripts/typecheck-low-memory.mjs`: passed. - Local source instance: verified a saved dependency wait stays neutral after reload; a saved GitHub error shows repair guidance, expands its original error, and opens the correct connection Permissions page. PostgreSQL and HTTP APIs were real. The GitHub denial was a controlled MCP fixture; no account or provider permission changed. - `pnpm build`: passed for all workspace packages, the Runner binary, server, CLI and UI. The server build and all server typecheck layers also passed after the review repair. - Additional safety checks passed for realistic repository arguments, active recovery, failed Retry preparation, and bounded provider diagnostics. - Review repair: 45 pure tests and 7 PostgreSQL cases passed. These cover both historical error codes, renamed apps, and company/provider/application identity boundaries. A second local browser check verified the renamed app, retained error detail, and correct Permissions destination. - Final review regressions: the dispatcher and metadata cases failed before the repair. Afterward, 25 focused cases and 59 projection/identity cases passed, including connection-key precedence. The native wait and app-only GitHub source also passed a local browser check. - The historical read adds at most one connection query per interaction read batch. It selects IDs and the provider key only and runs only for candidate permission failures. It adds no provider requests. - Full local `pnpm test:run` stopped at the chat suite's `afterAll` cleanup timeout. All 268 assertions in that group and the three prior chat groups passed. The failure is retained here; it is not counted as a full-suite pass. CI passed on `e8a692e1e2` after one workspace-startup rerun. Fresh CI is pending for the combined head. Its first workspace-a attempt timed out in the CLI company import/export test's child-server startup wait; 4,968 UI tests and 287 CLI tests passed in that job. The same shard passed on the preceding commit. The workspace shard passed on its one rerun. All other test, build and browser jobs passed. The [first failure](https://github.com/paperclipai/paperclip/actions/runs/42093823928/job/127478869941) is retained in the Actions history. Greptile reviewed the preceding head at 5/5. Fresh review is pending for the combined head. ## Risks The GitHub classifier matches the documented permission-denial phrase, not every 403. Rate limits, missing resources and other providers retain their existing error behavior. Historical normalization affects display only. A dependency cancellation needs an unstarted run and a supported queue receipt; current user gates, Retry preparation and recovery remain authoritative. No schema migration or credential changes. ## Model Used OpenAI GPT-6, through the Codex agent. The exact runtime model ID and context-window size are not exposed in this session. Used reasoning, repository tools, TypeScript, PostgreSQL, MCP fixtures and browser verification. The parallel agent-recovery commit records Anthropic Claude Opus 5.5 as co-author. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used with version and capability details - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in-PR using the bug template - [x] I have not referenced internal or instance-local Paperclip issues or links - [x] My branch name describes the change - [x] I have run the focused tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation - [x] I have considered and documented risks - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open findings - [x] I will address reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
ffd9591ab4 |
fix(recovery): preserve cancelled files after a reusable sandbox stops (#19305)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run writes its task files in a remote sandbox. The task
workspace records that run as its writer until the files are saved.
> - When a user stops or reassigns a run, the cancelled-workspace
preservation worker records the unsaved files as file recovery work.
> - A warm run (`keep_running`) that is not successful releases its
reusable sandbox through stop-and-retain. That release can finish before
the worker sees a terminal coordinator.
> - The worker accepted only an active lease or a released ephemeral
lease. It ignored the released reusable lease, so it never recorded the
file recovery.
> - The workspace kept the ended writer's hold. Each successor run
waited, then failed with "The workspace still has an unfinished writer
or save operation" after the one-hour cutoff.
> - This pull request accepts a released reusable lease that has a
confirmed stop-and-retain receipt as a preservation source.
> - The benefit is that the stopped run's files become recovery work,
and the task is not stranded behind a hold that nothing settles.
## Linked Issues or Issue Description
Refs #18820 (preserves files when Stop finds no live controller). Refs
#17677 (schedules cancelled workspace preservation). Refs #19285 (parks
a workspace wait behind an ended writer).
**What happened?**
A board user reassigned a task during a warm native run. Stop cancelled
the run with `issue_reassigned`, and the Stop acknowledgement was
recorded. The executor settled the coordinator to `terminal_failure`
with no owner. About eight seconds later, terminal release stopped and
retained the reusable Daytona sandbox. The lease became `released` with
`leasePolicy: reuse_by_environment`, a `stopped` termination receipt and
a stop-and-retain receipt. The task workspace stayed `dirty` with
`writerRunId` set to the cancelled run. No file recovery was recorded.
The new assignee's runs waited as `workspace_busy` for one hour. Then
each run failed `setup_failed`, and the task became blocked.
The heartbeat run row showed `nativePhase: "observed"`. That value is
stale: the checkpoint persist wrote it during teardown. The coordinator
itself was `terminal_failure`: the run's execution projection was
`recovery_needed` with no matching recovery action.
**Expected behavior**
The preservation worker records the stopped run's prepared files as file
recovery work. The lease keeps its release status and receipts. No
second stop and no new provider turn start.
**Steps to reproduce**
1. Cancel a native run with an acknowledged run-only Stop. Its remote
workspace reference is `prepared`, and the coordinator is
`terminal_failure` with no owner.
2. Release its reusable lease through stop-and-retain (`released`,
`reuse_by_environment`, confirmed stop receipt).
3. Run the cancelled-workspace preservation worker or the startup scan.
4. On master, the run has no `workspaceRestoreRecovery`, and no recovery
action exists.
## What Changed
-
`server/src/services/native-runtime/native-failed-workspace-recovery.ts`:
add one source check, `cancelledWorkspaceSource`, for cancelled
preservation. The worker check and the startup scan both use it. It
keeps the active and ephemeral cases. It adds a released reusable lease
only when the stop-and-retain receipt is confirmed
(`hasConfirmedSandboxStopAndRetain`) and `preparedRemoteWorkspaceSource`
proves the exact remote-workspace binding. This is the same proof the
restore recorder uses. Native-sync runs have no such proof and stay
ineligible. An ineligible source causes no run write or notification.
- The restore recorder (`preserveLegacyWorkspaceRestoreSources`) already
accepts this source. It checks the confirmed receipt and the exact
prepared workspace binding, and it keeps the lease status, policy and
receipts. This pull request does not change it.
- `native-cancelled-workspace-preservation.test.ts`: add the incident
shape as a regression for the worker and the startup scan, and add
safety cases.
- `doc/execution-semantics.md`: document the stopped reusable source and
the writer hold until recovery.
## Verification
From `server`:
```sh
npx vitest run src/services/native-runtime/native-cancelled-workspace-preservation.test.ts
```
- Red before the fix: the two new regression cases failed (`expected {
nativeCancellation: … } to match object { …, workspaceRestoreRecovery: …
}`). 30 other cases passed.
- Green after the fix: 36 passed.
- Review follow-up: the cases `invalid-receipt`, `expired-lease` and
`native-sync` failed on the previous head
|