mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-13 15:55:46 +01:00
e507a07bcf3634cd52c5535203ccaf9e49245966
230
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 DottaandClaude Opus 5.5
|
fa3cd6f330 |
fix(native): end provider startup rejections with Retry instead of an ownership hold (#20808)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The Paperclip Runner starts Codex for native runs. On a remote
sandbox such as Daytona, the sandbox is the OS boundary.
> - A Codex planning run on Daytona failed in `session.open`. Codex's
read-only sandbox helper could not set up loopback in the container
(`bwrap: loopback: Failed RTM_NEWADDR`), and Codex loads `AGENTS.md`
through that helper.
> - The server treated that definite provider rejection as an
interrupted transport. Its same-run bootstrap retry went through remote
restart recovery and stopped on `ambiguous_remote_recovery_evidence`.
The task showed "Recovering saved work…" and no Retry, although no
provider work was done.
> - This pull request gives such startup rejections a named, permanent
outcome with a working Retry, and pins ask and standard runs to the
external sandbox policy.
> - The benefit is that a provider startup refusal ends as a clear
failure with Retry, never as an ownership hold.
## Linked Issues or Issue Description
**What happened?**
A Codex planning run on a Daytona sandbox failed in `session.open` with
`error creating thread: failed to load AGENTS.md instructions ... bwrap:
loopback: Failed RTM_NEWADDR: Operation not permitted`. The server
recorded `native_session_interrupted` with recovery mode
`bootstrap_retry`. Thirty seconds later, remote restart recovery blocked
the run with `native_restart_recovery_blocked` /
`ambiguous_remote_recovery_evidence`. No server restart happened. All
default-mode Codex runs on the same image succeeded that day; only the
planning run failed.
**Expected behavior**
A provider startup failure with no provider work must end as a clear
failure with a working Retry, or get a bounded automatic retry that also
ends with Retry. It must never become an ambiguous-ownership hold.
**Steps to reproduce**
1. `cd server && npx vitest run
src/services/native-runtime/native-startup-rejected-retry.integration.test.ts`.
On master the deterministic case is `retryable_failure` /
`native_session_interrupted`.
2. `npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t "provider
itself rejected"`. On master the new code is not produced.
**Paperclip version or commit**
master at
|
||
|
Nicky Leach
|
4d141cdf64 | Carry plugin task refusals and batch project reads (#20762) | ||
|
DottaandClaude
|
f25fee3532 |
fix(native): hash runner artifacts and run archive tar off the event loop (#20722)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - One server process prepares native runs, renews every controller lease on timers, and serves every request > - On a run path, the server hashed the 31 MB runner binary with `readFileSync` (more than once per run), hashed provider packs and harness backups synchronously, and ran `tar` with `execFileSync` (timeouts up to 60 s, one with no timeout) > - On a slow disk, each of these calls freezes the whole process. A 103 s stall on a hosted instance lapsed every lease. The last event before that stall was immediately before a synchronous read and hash of the runner binary (see #20694 and #20709) > - This pull request moves those reads, hashes and archive commands off the event loop. It computes the runner binary digest once per boot > - The benefit is that a slow disk delays only the run that waits for it. Turns also start faster, because the runner binary is no longer re-read for every run ## Linked Issues or Issue Description Refs #20694, #20709 **What happened?** On a hosted instance, the server process stopped for 103 s without a restart. Every controller lease lapsed, and two tasks needed a manual Retry. The last run event before the gap was immediately before `matchesControllerRunnerArtifact`, which does `createHash("sha256").update(readFileSync(controllerRunnerBinary))`. Four workspace restores were running at the same time, and file I/O was very slow (a 43 s archive hash, 20 s snapshot decrypts). The trigger is not proven. But each synchronous call below blocks the event loop for as long as the disk is slow. **Expected behavior** File hashing and archive commands on a run path must not block the event loop. A slow disk must delay only the run that waits for it. **Steps to reproduce** 1. Make a remote runner binary slow to read (the test uses a FIFO whose writer waits), or make `tar` slow (the test puts a `tar` wrapper that sleeps first on `PATH`). 2. Call `resolveRemotePaperclipRunnerBinary`, `stageRemoteRunnerDirectory`, or `syncRemoteRunnerDirectoryOut` while a 10 ms timer runs. 3. Before this change, the timer stops for the whole slow step (814 ms, 1,764 ms and 3,379 ms). After this change, it keeps firing. ## What Changed - `server/src/services/native-runtime/off-loop-file-work.ts` (new): - `sha256FileStreamed` - `cachedFileSha256`: a streamed digest, reused while the file's device, inode, size, mtime and ctime do not change - `binaryTargetFromHeader`: reads 64 KiB; reads the rest of the file only when the header cannot answer, so the result equals `runnerBinaryTarget(readFileSync(path))` - `sha256DirectoryTreeStreamed` - `execFileBuffered`: `execFile` with stdin input, `maxBuffer`, `timeout` and `env`. It rejects on a non-zero exit, a timeout or a `maxBuffer` overflow, like `execFileSync`. - `native-session-executor.ts`: - `matchesControllerRunnerArtifact` uses `cachedFileSha256`. Each boot reads the binary once, not once or twice for every run. - The remote Claude daemon target is read from the binary header. - The bundled remote runner binary is checked by `verifiedBundledRemoteRunnerBinary`: the same checks and errors, with a cached digest. - Provider-pack upload verification uses the new `verifyRemoteProviderPackManifest`. It runs the same checks, in the same order, with the same errors. One generator drives both the sync and the streamed readers. - Harness backup build and verify on run paths use streamed digests (`buildNativeHarnessBackupManifestStreamed`, `verifyNativeHarnessBackupStreamed`). The digest format does not change. - `stageRemoteRunnerDirectory` (3 calls), `assertSafeRemoteCheckpointArchive` (3 calls) and the checkpoint extract use `execFileBuffered`, with the same arguments, timeouts and buffer limits. `stageRemoteRunnerFile` uses `readFile`. - `packages/paperclip-runner/src/live/runner-binary.ts`: the default daemon lookup (`resolvePackagedRunnerBinary`, used by the local and remote resolvers) classifies the daemon with `runnerBinaryFileTarget`. It reads the 64 KiB header, not the whole file. - `packages/paperclip-runner/src/live/bundled-remote-provider-pack.ts`: the release-manifest checks now live in one helper, `bundledRemoteRunnerRelease`. The sync `bundledRemoteRunnerBinary` and the server's streamed check both use it. - `native-codex-runner.ts`: - `resolveRemotePaperclipRunnerBinary` reads only the header. - The local `runnerDigest` uses `cachedFileSha256`. - New `verifiedBundledRemoteRunnerBinary`. **Ordering and atomicity.** I checked each converted call for code that relied on the sync call being atomic with respect to other runs: - **Checkpoint copy-back.** It extracts into a private `mkdtemp` directory. The replace sequence (`existsSync`, `rename`, `rename`, `rm`) stays synchronous, so no other run can see a half-installed checkpoint. This function already awaited a remote command before the extract, so this change adds no new interleaving with the target. - **Harness backup verify then restore.** The verify was sync, but the restore after it was already async. The window between them existed before. One owner serializes the turns of a session, so no other turn publishes a backup for that session while a turn verifies it. - **Staging and provider-pack hashing.** These read build-owned, immutable trees, or the private staging copies of a run. ## Verification - Red/green: `pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/off-loop-file-work.test.ts` - With `native-session-executor.ts` and `native-codex-runner.ts` from `origin/master`: 3 failed. The longest timer gaps were 814 ms (runner binary target read), 1,764 ms (directory staging tar) and 3,379 ms (checkpoint list, size check and extract). - With this change: 6 passed. The gaps stay under 400 ms. The same file also checks the digest cache, the header fallback, and the `execFileSync`-like errors (exit code, timeout kill, `ERR_CHILD_PROCESS_STDIO_MAXBUFFER`). - `native-session-executor.test.ts`: - The streamed provider-pack verifier returns the same manifest as the sync reader, and rejects a changed asset tree with the same error. - The streamed harness backup manifest equals the sync one. - The streamed verify agrees with the sync verify, for a valid backup and for a corrupt one. - Run together: `native-session-executor`, `off-loop-file-work`, `native-codex-runner`, `setup-readiness`, `antigravity-connection-probe` (1,653 passed). `run-completion.test.ts` passed all 153 tests. Its `afterAll` embedded-Postgres cleanup timed out locally, which is not related to this change; CI runs it. - `pnpm --filter @paperclipai/server exec tsc --noEmit` passes. ## Risks - **Turn start.** This change adds no wait before `turn.start`; it removes work. The first run of a boot streams the runner binary once. Later runs reuse the digest after one `stat`. - **Cache staleness.** The cache keys on device, inode, size, mtime and ctime. An in-place rewrite changes mtime and ctime, and a replaced file changes its inode, so a changed binary is hashed again. - **stderr.** `execFileSync` wrote a child's stderr to the server's stderr. `execFileBuffered` captures it. Tar warnings no longer appear in server logs; errors and exit codes are unchanged. - **Persisted state.** No persisted shape changes. Old saved state and new code (and the reverse) read the same manifests and digests. - **Not converted in this PR:** small metadata syscalls (`existsSync`, `lstatSync`, `readdirSync`, small JSON reads and writes, renames) in the native state machines. Some of these rely on running synchronously. Also not converted: the sync harness backup verify inside the sync restart-authority checks, and the sync stamp verify in `native-harness-backup-storage.ts`. Both are listed for a follow-up. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude Code with tool use, shell, and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes (code comments carry the contract) - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude
|
08ff0828d6 |
fix(native): offer Retry at once when the Runner refuses a turn start on a closed session (#20729)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native ACPX run (for example Claude) starts each turn with `turn.start`. When a start is cut off, the Runner reads the sidecar's dispatch journal (#20675). > - If the prompt never left, the Runner keeps the session reopenable, and the same-run resume sends the prompt once. If the prompt may have left, the Runner closes the session, because a resend could duplicate it. > - A closed session cannot reopen for the same run. Every same-run resume got `ACPX session is not open`. The server treated that as a generic interruption and spent three attempts (about 95 s) before the board offered Retry. > - This pull request gives that refusal a closed class in the Runner and a permanent, named failure code in the server. > - The benefit is that the board offers a working Retry on the first refusal, while every recoverable case keeps recovering by itself. ## Linked Issues or Issue Description **What happened?** Restart-chaos wave `w-20261111T011704-s9107`, task t05 (local, claude, workload C1, fault TM2). A freeze cut off a turn start, and the Runner saved the session `closed`. Three same-run resumes each failed with `PRP command turn.start failed: … "message":"ACPX session is not open"`. Recovery stopped with `native_session_retry_exhausted` at 01:25:14, and the board then offered Retry. The Retry run succeeded. **Expected behavior** When the Runner proves the session is closed for this run, the run stops at the first refusal with a named failure, and the board offers a Retry that works. A start that the Runner proved never reached the provider still recovers by itself. **Steps to reproduce** 1. `cd server && npx vitest run src/services/native-runtime/native-closed-session-retry.integration.test.ts` 2. On master, "offers Retry on the first refusal, and Retry starts a new run" fails: the coordinator is `retryable_failure` / `native_session_interrupted` (a same-run retry in 30 s). Refs #20675, #20720 ## What Changed - Runner (`durable/mod.rs`, `acpx_provider_backend.rs`): new closed class `provider_session_closed`. The Runner sets it on these refusals: - a turn start that saves the session `closed` (`ACPX turn start failed closed: …`), but not the rejected-checkpoint case, which stays `prepared` and reopens; - `ACPX session is not open` when the saved lifecycle is `closed` for this run; - the saved turn-end-restart refusal for this run (asserted in `acpx_turn_end_restart`). A session that waits for provider-exit proof can reopen after a restart. Its refusal gets no class. - Server (`native-session-executor.ts`): a `turn.start` failure with that class maps to the new permanent code `native_provider_session_closed`. The server reads the class from the authenticated command result, and from the command-failure text if the error was rewrapped. `nativeSessionFailureDisposition` treats the code as permanent, so the first refusal ends the run as `terminal_failure` with a board owner and a plain next action. - Docs: `packages/paperclip-runner/docs/durable-recovery.md` states the class and the older-Runner behavior. - Tests: - `native-closed-session-retry.integration.test.ts` (new, embedded Postgres) drives `executePaperclipNativeSession` with the Runner refusal. On the first refusal, the board shows `canRetry: true`. `requestTaskRetry` is accepted, and `resumeTaskRetry` starts exactly one new run with a fresh provider session. A start proved undispatched stays `retryable_failure`. - Unit tests in `native-session-executor.test.ts`: both closed refusals, a rewrapped error, an undispatched start, an unclassified older-Runner refusal, and a class on another command. - Rust `acpx_cut_turn_start`: a start cut off before dispatch has no class. A start cut off after dispatch, and the later refusal, carry `provider_session_closed`. ## Verification - Red on master, green here: step 1 above. - `npx vitest run src/services/native-runtime/native-session-executor.test.ts`: 1358/1358 pass. - `cargo test -p paperclip-runner-core --test acpx_park_preparation` (26 pass), `--lib acpx` (94 pass), `--test acpx_provider_turns --test acpx_provider_session --test acpx_warm_attachment` (all pass), `cargo fmt --check` clean. - `npx tsc --noEmit -p server/tsconfig.json`: clean. - Not run: a chaos wave. ## Risks - This changes the Rust Runner. Daytona needs a new runner image before remote runs get the class. Until then, an older Runner sends the refusal without the class, and the server keeps today's bounded same-run retries. That path is unchanged and ends with a working Retry. The server does not trust the text alone, because an older Runner sends the same text for a session that can still reopen. - The class is set only where the Runner's saved state proves that no later start of this run can reopen the session. A false class would cost one automatic resume and show Retry earlier. It never replays provider work. - No persisted shape changes. The class is an optional field of a failed command result. Old saved state resumes unchanged. Nothing new runs before `turn.start`. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude (Anthropic), model ID `claude-opus-5-5`, run in Claude Code with tool use (shell, file edit) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude
|
200c6e40e2 |
fix(native): restart a bootstrap whose Runner suspended before the provider opened (#20691)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - When a native run's first provider start fails before any session opens, Paperclip retries the bootstrap on the same run. Durable controller evidence proves that no provider session or provider event exists. > - A remote sandbox keeps the Runner state inside the workspace. A replacement sandbox gets that state back when the saved workspace is restored. > - When the provider refused `session.open` (for example an Anthropic 429), the Runner still suspended cleanly. Its state says `suspended`, but the provider state has no session identity. > - The same-run restart handled a Runner that never suspended. It did not handle a Runner that suspended before the provider opened a session. It failed with `runner_harness_state_mismatch: provider_identity_incomplete (provider_not_suspended)`, and the task showed "Recovery needed" without Retry. > - This pull request lets that proven restart set the unopened session aside and start the provider again. > - The benefit is that a rate-limited provider start continues by itself after the limit resets. ## Linked Issues or Issue Description Refs #20627 and #20683 (other restore paths). This PR changes only the same-run bootstrap restart. **What happened?** Two ACPX Claude tasks on remote Daytona sandboxes, on staging: 1. The first attempt sent `session.open`. ACPX rejected it with `classification=provider_rate_limit` (Anthropic 429). The Runner exited with `lifecycle=suspended`. 2. Paperclip recorded `recoveryMode: bootstrap_retry`, `providerSessionEstablished: false`, `providerEventsExist: false`, and a retry after the limit reset. 3. One hour later the retry ran as `bootstrap_incomplete` on the same run, in a new sandbox with the saved workspace restored. 4. Host preparation inspected the restored Runner state. The Runner was `suspended`, but the provider had no session identity. The inspection threw `provider_identity_incomplete (provider_not_suspended)` before the bootstrap restart branch ran. 5. The run failed. The task stopped at "Recovery needed" with only inspect actions. **Expected behavior** The restart sets the unopened session aside and opens a new provider session, as it already does for a Runner that never suspended. **Steps to reproduce** 1. Restart a remote run as `bootstrap_incomplete` with a proven bootstrap restart. 2. Give the sandbox a Runner state for this run with `lifecycle: suspended` and a provider state with no session identity. 3. Prepare the host. On `master` it throws `runner_harness_state_mismatch: provider_identity_incomplete`. The new `interrupted_bootstrap_unopened` cases of `bootstraps only an untouched provider session in a $outcome workspace lease` run these steps for a created and a resumed lease. **Paperclip version or commit** `master` at `7435abcc63`. ## What Changed - During host preparation, a Runner state that is suspended with no provider session identity returns the new incomplete reason `provider_unopened`, but only when all of these hold: - the run is restarting as `bootstrap_incomplete`; - the state belongs to this same run; - the controller already recorded the bootstrap restart proof (its durable journal shows no provider turn, and the old Runner is stopped or absent). - the provider state is the one saved before any session opened (`remoteProviderStateNeverOpened`): `prepared`, no session or identity, no turn in progress, no unconfirmed exit. Only ACPX and Codex-family states qualify. - The created-lease and resumed-lease restart branches share one check (`restartableBootstrapState`) and accept `provider_unopened` exactly like `not_suspended`. They move the old session aside and claim a new, untouched session root. - Every other case still throws `provider_identity_incomplete`: another run's state, a restart without the proof, a normal resume, and close checkpointing. ## Verification - `pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/native-session-executor.test.ts -t "bootstraps only an untouched provider session"`: 47 passed. - Red on `master`: the `created` and `resumed` `interrupted_bootstrap_unopened` cases fail with `runner_harness_state_mismatch: provider_identity_incomplete (identity_binding)`. Green with this change. - Fail-closed cases stay red-proof: `interrupted_bootstrap_unopened_unproven` (no restart proof) and `interrupted_bootstrap_unopened_other_run` (another run's state) still throw. - Full `native-session-executor.test.ts` and `pnpm --filter @paperclipai/server typecheck` pass. ## Risks - Low risk. The new branch needs the same proof that already authorizes a fresh bootstrap for a Runner that never suspended. The proof comes from the controller's durable journal, not from the sandbox. - The normal path is unchanged and has no added wait. - Not proven live: I did not confirm on staging that the controller recorded the restart proof for these runs. If it did not, the run still fails closed as before. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude
|
a1ec05a1a4 |
fix(native): read settled harness state again after a dropped sandbox connection (#20690)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native Runner keeps one provider conversation across the per-turn runs of a task. After each turn, the controller reads the settled Runner and provider state out of the sandbox and saves it as the harness backup. > - A per-turn sandbox is released after the turn. The harness backup is then the only copy of the provider session. > - On staging, one dropped sandbox connection (`write ECONNRESET`) during that read lost the backup. The run itself succeeded, because its result was already accepted. > - The next run of the task could not resume the conversation: `PRP provider resume state is unavailable`. When the task was waiting for a reply, the run stopped at "Recovery needed" with no Retry. A person had to send a message to start a new conversation. > - This pull request reads the settled state again when the sandbox connection drops during that read. > - The benefit is that one network reset no longer loses a task's conversation. ## Linked Issues or Issue Description Refs #20627 (warm backup restore when the source sandbox is idle) and #20683 (relaunch of a settled runner). This PR changes a different step: the backup capture at the end of a turn. **What happened?** A per-turn OpenCode task ran on a remote Daytona sandbox with reuse disabled. 1. Run A accepted its result. The Runner suspended. 2. The controller read the settled state out of the sandbox. The read failed with `remote checkpoint failed: write ECONNRESET`. `harness_state.backup.persist` and `session.checkpoint.persist` failed. 3. Run A still finished as `succeeded`, because the result was durable. The sandbox was released. No harness backup of turn A existed. 4. Run B started in a new sandbox. It failed in 0.2 s with `native_session_recovery_failed: Error: PRP provider resume state is unavailable`. When the prior turn yielded for a reply, the checkpoint policy does not allow a replacement session, so the task showed "Recovery needed" with only inspect actions. On staging this happened on 4 tasks in 48 hours. In all 3 predecessor runs that I could inspect, the backup capture failed. 2 of them show the `ECONNRESET` in the run log. No deploy happened between run A and run B in one case, so a deploy is not the cause. **Expected behavior** A transient connection drop while reading settled state does not lose the provider conversation. The read is repeated, and the backup is saved. **Steps to reproduce** 1. Run the first durable turn of a remote OpenCode session. 2. Make the sandbox `syncOut` of the backup directories fail once with `ECONNRESET`. 3. Call the settled checkpoint. On `master` it throws `write ECONNRESET`, and no backup is saved. The new test `uses the bundled Linux image authority for remote opencode-new-durable-reset without controller overrides` runs these steps. **Paperclip version or commit** `master` at `7435abcc63`. ## What Changed - The settled-state capture (inspect plus `syncOut` of each persistence directory into a pending directory) is now one step that can repeat. The step writes nothing durable, so it is safe to repeat. - A dropped, reset or refused sandbox connection repeats the step, at most 3 attempts, with 1 s and 2 s delays. Each repeat clears the pending directory and inspects the live state again, so the copy always matches the inspected identity. - Any other failure fails at once, as before: a state mismatch, a missing directory, an unsafe archive, or a TLS failure. - The durable backup upload and the local publish run only after a complete capture. They did not change. - `isTransientRemoteCheckpointReadFailure` classifies the error with the existing `classifyRemoteConnectionError`, plus a message match for SDK errors that lose their `code`. - The first completeness inspection and its visibility-window re-reads use the same bounded retry (`retryTransientRemoteCheckpointRead`). A dropped provider-state read keeps its `provider_state_unreadable` refusal for every other caller, but the checkpoint retry looks through it to the dropped read. Undici `fetch failed` and `no response returned` also count as drops. ## Verification - `pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/native-session-executor.test.ts`: 1343 passed. - Red on `master`: the new `opencode-new-durable-reset` case fails with `Error: write ECONNRESET`. Green with this change. - The `isTransientRemoteCheckpointReadFailure` test checks that state mismatches, unsafe archives and TLS failures do not repeat. - `pnpm --filter @paperclipai/server typecheck` passes. ## Risks - Low risk. The normal path has no added wait. A repeat adds at most 3 s, and only after a connection failure. - A repeated read inspects the live state again. If the state changed between attempts, the existing identity checks still refuse it. - A connection that stays down for all 3 attempts fails the same way as before. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude
|
2eede8b318 |
fix(native): give a relaunched settled runner its admitted provider pack (#20683)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - After a server restart or a same-run retry, a remote native run reattaches to its runner in the sandbox (`reattach_remote_runner`). > - When the controller saved the runner artifact, the executor prepares that run as an adoption. It skips provider-pack selection, because a live adopted runner keeps its launched provider. > - The reattach probe can instead find the runner settled and exited (`verifyRemoteRunnerOrSettledExit`). Then nothing is adopted, and the transport relaunches the runner from its durable state. That relaunch must start the provider again, so it needs the provider-pack launch inputs. > - The executor did not give them. OpenCode failed with "OpenCode executable is missing from the provider pack". ACPX failed with "ACPX launch profile omitted its provider-pack authority or verified artifact digests". Both ended the run. > - This pull request selects the saved (admitted) provider-pack authority on that settled-exit relaunch and verifies it as an exact installed identity, as an ordinary reattach does. > - The benefit is that a restart or a same-run retry can resume an exited runner with the same provider pack, instead of failing. ## Linked Issues or Issue Description **What happened?** On a hosted staging instance, two OpenCode runs and one ACPX (Claude) run were interrupted by a deploy (SIGTERM) or ended a turn without a result. The server scheduled a same-run retry that reattached to the existing remote runner. The runner had exited cleanly (`runner_remote_process_exited lifecycle=suspended`). The retry then failed within three seconds with `runner_remote_provider_artifact_incompatible`. **Steps to reproduce** 1. Prepare a remote OpenCode or ACPX run with a persisted runner artifact (`controllerPersistence.runnerArtifact`). 2. Reattach with `restartRecovery.kind = "reattach_remote_runner"`, where the runner process is not running and its state is `suspended` and settled. 3. Observe that the transport options have no `adoptExistingRunner`, no `providerPackAuthorityDigest` and no `opencodeCommand` / `acpxSidecarSha256`. **Expected behavior** The relaunch gets the admitted provider pack's launch inputs, or it fails with a clear pack-identity error. ## What Changed - `createRunnerdBackend` (native session executor): when a persisted-runner reattachment finds no live runner to adopt, select the saved provider-pack authority (`reattach: true`), so `expectedProviderPackManifest` is the admitted pack. - `prepareRemoteRunnerAssets` no longer returns early for that relaunch. It verifies the installed pack as an exact identity and sets the active pack root, the same path as a reattach without a persisted artifact. - A live adopted runner is unchanged: it still skips pack selection. - The test `uses the bundled Linux image authority for remote opencode|claude-x64` now also covers the settled-exit relaunch for OpenCode and ACPX. ## Verification - RED on `origin/master` (`c444da7d05`): the extended test fails for `opencode` and `claude-x64` (the relaunch options have no provider-pack inputs). - GREEN with this change: `pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/native-session-executor.test.ts -t "uses the bundled Linux image authority"` (14 passed). - `pnpm --filter @paperclipai/server typecheck` passes. - `pnpm verify`: see the result below. ## Risks - Upgrade safety: the relaunch reads the saved authority, so a newer controller does not select a newer pack for an older session. It requires an exact installed identity. If the sandbox no longer has that pack, the relaunch fails with a pack-identity error, and the existing Retry path applies, as before. - No change to a live adoption, to fresh runs or to the normal path before `turn.start`. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, read-only staging API reads). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
fd6e71a7ff |
fix(native): restore a warm backup when its source sandbox is idle (#20627)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs on remote sandboxes keep a warm sandbox and a verified harness backup so the next run continues the same provider session > - A run that starts while the previous run's workspace export is still saving gets a new (created) sandbox lease, because reuse skips an exporting sandbox > - When that run prepares the harness, it finds a backup but no recorded reusable lease, and it fails closed with `runner_harness_state_mismatch: backup_without_reusable_lease` > - #20278 accepted a destroyed source and a settled restore hold as retired backup sources, but not an idle warm source that is still `retained` for reuse > - This pull request accepts an idle warm source under the same proofs, claims it so it is never resumed again, and covers the resumed lease that the failed run leaves behind > - The benefit is that these recoveries continue the same session automatically instead of blocking the task ## Linked Issues or Issue Description Refs #20278 **What happened?** A monitor wake started about 90 seconds after the previous run on the same task finished. The previous run's warm sandbox lease was `retained` with policy `reuse_by_environment`, but its workspace export was still saving. Reuse skipped it and created a new sandbox. Workspace realization waited for the export (about 136 s) and restored the saved files. Then harness preparation found the previous run's backup (its lease stamp names this exact manifest, and no provider work followed it). The source lease was not destroyed and was not a restore hold, so the run failed with `native_session_recovery_failed: Error: runner_harness_state_mismatch: backup_without_reusable_lease` and the task was blocked. **What did you expect?** The run continues the same provider session. The idle source sandbox holds exactly the backed-up state, so restoring the backup into the new sandbox is safe. **Steps to reproduce** 1. Finish a warm native run on a remote sandbox. Its workspace export is still saving. 2. Start the next run on the same task before the export settles. 3. Reuse skips the exporting sandbox and creates a new lease. Harness preparation fails with `backup_without_reusable_lease`. 4. A retry resumes that new lease, which has no harness state, and fails with `resumed_unavailable`. ## What Changed - `native-retired-backup-source.ts`: an idle warm source is a third retired backup source. It must be `released` or `retained` with policy `reuse_by_environment`, its own run must be terminal and not the current run, and it must have no cleanup in flight, no restore hold or pending shutdown settlement, and no pending native workspace export. The backup stamp, single-owner chain, and later-provider-work checks still apply. - Before the restore, the idle source is claimed with a compare-and-swap on the checked row. The claim moves its `reusableSandboxLease` scope into a `reusableSandboxLeaseSuperseded` receipt. Reuse never resumes that older state again, and the next acquisition treats it as obsolete and destroys it. If a run resumed or changed the source first, the claim matches no row and the backup is not restored. - `environments.ts`: a cross-run reuse handoff now requires that the source row has no superseded receipt. The supersede claim and the handoff compete on the same row, so an acquisition that selected the source before the claim cannot resume it afterward. - The proof now rejects a backup whose source is the target allocation itself (`targetProviderLeaseId`). - `native-session-executor.ts`: a resumed lease whose sandbox has no harness state at all (`unavailable`) restores the backup under the same proof. This recovers the lease that the failed run left behind, so rows saved by older releases recover after deploy without a migration. - `doc/execution-semantics.md`: documents the idle-source rule, the supersede claim, and the resumed stateless lease. ## Verification - Red before the fix, green after: - `native-session-executor.test.ts` "restores a warm backup into a created lease only when an idle warm source finished its run and its export" failed with `backup_without_reusable_lease`. - "... a resumed lease never held the session and the warm source is idle" failed with `resumed_unavailable`. - `native-retired-backup-source.integration.test.ts` (real Postgres) "authorizes an idle retained/released warm source and claims it as superseded" failed (returned false). - Integration test "refuses a reuse handoff of a source superseded while its provider resume was pending" fails without the handoff guard and passes with it. - Safety rows stay refused: source run still running, workspace export pending, cleanup in flight, restore into its own allocation, stamp names another checkpoint, provider work after the backup, two host leases for one allocation, claim lost to another run, and a resumed lease that lost its own state. Existing destroyed-source and restore-hold rows are unchanged. - `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts src/services/native-runtime/native-retired-backup-source.integration.test.ts src/services/native-runtime/remote-warm-session-recovery.test.ts` passes (1408 tests). - `npx tsc --noEmit -p server/tsconfig.json` passes. - `pnpm verify` exits 2 because a global CI input (`doc/execution-semantics.md`) changed. Required CI supplies the broad coverage. ## Risks - Previous-release story: no persisted shape changes. Lease rows saved by earlier releases carry forward. The new `reusableSandboxLeaseSuperseded` metadata key is additive. - The idle source is destroyed at the next acquisition for its workspace. Its state is the verified backup, which the restored lease now holds, so no work is lost. - Not covered: when the source cannot be proven retired or idle (for example a live owner or later provider work), the run still fails closed with the same named code. An automatic fresh-session fallback for that case is not in this PR. - The root trigger (reuse skips a sandbox while its export saves, then waits for that export) is unchanged. This PR does not change the reuse lookup. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Nicky LeachandPaperclip
|
19e938415a |
fix(runtime): omit Codex refresh handoffs from backups (#20614)
## Thinking Path > - Managed ACPX Codex retains its final refresh after provider exit so Core can save it. > - Shutdown saves a checkpoint before Core removes the live credential handoff. > - Checkpoint profiles must exclude the handoff while retaining resumable session history. ## Linked Issues or Issue Description Follow-up to #20580 and its credential-backup review finding. Refs #20457. ## What Changed - Exclude `auth-refresh.json` and `auth-refresh.json.tmp` from the shared Codex checkpoint profile, covering native and ACPX homes. - Extend real checkpoint-copy coverage to ACPX Codex, both handoff filenames, and native Codex. Keep the live files and session history intact. ## Verification - A real ACPX checkpoint-copy test failed before the exclusion fix; Grok passed. - Five targeted real-copy/profile tests pass, including after applying the change to current master. - `git diff --check` and local secret/PII diff review pass. - Full regression coverage runs in GitHub CI at the maintainer's request. No live provider credentials were used. ## Risks - Two entries added to an existing exclusion list; no runtime lifecycle, schema, persisted identity, or session format changes. Saved sessions remain readable. Existing backup restoration uses the same exclusion profile. - Provider credential cleanup still follows final writeback; excluding a backup entry does not delete the live handoff. ## Model Used - OpenAI GPT-6 through Codex, with tool-assisted implementation and verification. The runtime does not expose a more specific model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Nicky Leach
|
17136a6c25 | fix(runtime): persist managed Runner credential refreshes across execution targets (#20580) | ||
|
DottaandClaude Opus 5.5
|
be0d100a8b |
fix(native): settle cleanup-gate refusals after a completed turn without a dead-end Retry (#20572)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs execute a provider turn, then close the provider session
and prove it stopped
> - When close cannot prove that the old runner, provider and tools
stopped, the runner holds a cleanup gate
(`native_session_cleanup_quarantined`) that refuses the next session on
that task
> - A run finished its provider turn and saved its work. A server
restart had left one tool callback without a saved result, so close
could not prove the drain, and the gate refused the run's own same-run
recovery 30 seconds later
> - The task then showed "Couldn't finish this attempt" with a Retry
button, but the retry route refuses this error code with 409. The saved
error text told an operator to act
> - This pull request makes the task projection, the execution blocker,
the executor and the retry route agree, and replaces the operator text
with plain wording
> - The benefit is that finished work no longer looks like a failed
attempt, and the UI never offers a Retry that the server refuses
## Linked Issues or Issue Description
**What happened?**
A remote native run completed its provider turn and saved its comments
and documents. A server restart during the turn left one semantic tool
callback without a saved result. Session close then reported
`native_session_settlement_incomplete` (runner suspended, provider drain
unproven), and the runner kept an operator-required cleanup gate. The
same-run disposition recovery was refused at admission, and the run
ended `failed` with `native_session_cleanup_quarantined`.
**Expected behavior**
The task should show its real state (here, waiting on a child task). It
should not show a failure banner or a Retry that the server refuses. No
text should ask an administrator to act.
**Steps to reproduce**
1. Start a remote native run and restart the server while a tool
callback runs.
2. Let the resumed turn complete without a semantic result, so a
same-run recovery is scheduled.
3. Observe the refusal at admission, the "Couldn't finish this attempt"
banner, and a Retry that returns 409.
**Paperclip version or commit**
master at
|
||
|
Nicky LeachandPaperclip
|
2e3cbe57a4 |
fix(native): preserve parked Runners across controller restarts (#20586)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native execution parks a released sandbox Runner for a later turn. > - The warm idle timer detaches Core and leaves Runner retention to the host. > - Controller shutdown instead stops recently parked Runners through checkpoint cleanup. > - This makes reuse depend on whether the idle timer fired before shutdown. > - This pull request uses the same detach operation for idle expiry and shutdown. > - The benefit is consistent Runner retention without a new retention or recovery mechanism. ## Linked Issues or Issue Description Fixes: #20012. Related: #20005 unified cached and adopted process reuse. Closed #19975 included this shutdown change within a broader retention proposal. ## What Changed - Detach confirmed released entries during controller shutdown. Keep busy, unreleased, and close-marked entries on the existing cleanup path. - Share the detach operation with idle expiry and cached-owner handoff. - Record that a local handle has started detaching. Later cleanup cannot fall back to stopping its Runner after a partial detach failure. - Preserve explicit environment deletion, release proof, admission, and process-reuse checks. - Add Codex and Claude restart, detach-failure, and environment-deletion regressions. Add real Runner adoption cases after the restart sweep, including managed GitHub rebinding. - Document the lifetime rule and make the shutdown warning cover both detach and checkpoint failures. ## Verification - Final source revision: `e4bb6ecbd7`. Local environment: macOS arm64, Node 26.12.0. - New controller-restart and detach-failure tests fail against unchanged master: 4 expected failures. Old shutdown calls close or reports success instead of reporting detach failure. - Focused executor tests: 22 passed. - Full native session executor suite: 1,328 passed. - `cd server && node scripts/typecheck-layers.mjs`: passed all source and test layers. - `node scripts/check-module-boundaries.mjs` and `git diff --check`: passed. - Manual diff review and added-line credential/PII scan: passed. - `cd server && node ../node_modules/vitest/vitest.mjs run src/services/native-runtime/plugin-runner-release-resume.integration.test.ts --bail=1 --reporter=verbose`: 14 passed on this revision. This includes new Codex and Claude restart cases and managed GitHub rebinding. - Failure history: an earlier concurrent run had 4 passes and 10 failures. Its first failure was the existing Claude cached case exceeding 60 seconds. Later cases encountered a leftover prototype spy, recursive calls, and startup timeouts. No source changed before the passing full rerun. A clean worktree of unchanged master (`e01fe7f84a`) passed all 11 original cases. The first timeout's exact cause remains unknown; the later pass does not establish that cause. - Standard Greptile review: 5/5 on `e4bb6ecbd7bd965b5f56507d2c42c580e8a1d547`, with no actionable findings. - `node scripts/local-verify.mjs --explain` returned exit 2 because the execution-semantics document requires broad CI. Required CI will supply broad typecheck, test, and build coverage. - A hosted deployment and an actual controller process replacement have not been run. The added integration cases exercise the shutdown sweep and a new adopting transport against a real Runner. ## Risks - SIGINT and SIGTERM preserve confirmed parked Runners, including ordinary service stops. The host still owns eviction and retention. - Detach failure retains a fenced local owner and reports failure. Core does not infer permission to stop the Runner from that failure. - Reuse remains conditional on current admission, process configuration, and host availability. - No persisted schema, checkpoint format, launch identity, or migration changes. State from the previous release uses the existing adoption or verified restore paths. ## Model Used - OpenAI Codex, GPT-6, with reasoning, repository inspection, tool use, and local test execution. The exact deployment identifier and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
c0593b69f1 |
fix(accounting): close a remote-Runner turn completed across a session restore as a lower bound (#20524)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runs settle their charge once the final usage receipt is complete. > - #20517 fixed one case for the local Runner. If a restart stops the provider before a turn's final usage report, and the restored Runner then reports that turn completed, the turn now closes as a lower bound instead of waiting 10 minutes. > - Remote Runners deliver usage through a separate producer-owned receiver. That receiver dropped `session.resumed`, so the same turn there still waited for the 10-minute closure timeout. > - This pull request applies the #20517 rule to that receiver. > - The benefit is that remote ACPX and OpenCode runs settle promptly with an honest lower-bound label after a mid-turn restore. ## Linked Issues or Issue Description Refs #20517. **What happened?** After #20517, `session.resumed` reached turn accounting only through the local Runner's fallback recorder. The remote producer receiver (`native-usage-delivery.ts`) and the session-authority filter (`isNativeUsageAccountingEvent`) both dropped it. A remote ACPX or OpenCode turn that was open across a session restore, and then reported terminal without a complete report, left the receipt incomplete until the closure timeout. **Expected behavior** Same rule as the local Runner. Such a turn closes as a lower bound right away. A complete report that arrives after the restore is still used, and a turn that starts after the restore stays pending until its own report is complete. **Steps to reproduce** `native-usage-delivery.integration.test.ts > a turn restored mid-turn on a remote Runner > closes the turn as a lower bound, also after the controller restarts between restore and terminal` fails on master: the receipt stays `accountingReceiptReady: false`. ## What Changed - **Receiver (`createNativeUsageReceiver`).** It now accepts `session.resumed` for turn-reporting providers (ACPX, OpenCode) while a turn is open. It journals the restore as a receipt like any other producer event, so turn accounting marks the open turn restored. A replacement controller rebuilds the same state from that journal. - **Restores never fail delivery.** A restore from another run or source, or one with no open turn, is ignored, not rejected. An unacknowledged lifecycle event would otherwise replay without end. - **Session-authority filter.** `acceptAccountingEvent` now uses a new `isNativeUsageDeliveryEvent`: the usage events plus `session.resumed`. A restore for another run is ignored there too. - **Recovery routing is unchanged.** `acceptRecoveryAccountingEvent` and `onRecoveryAccountingEvent` keep the usage-only `isNativeUsageAccountingEvent`, so a source run's restore never requires a proven recovery source. - **Turn accounting** gets a `hasOpenTurn()` helper. - **Docs:** `doc/SPEC-implementation.md` now covers both Runners. ## Verification - `native-usage-delivery.integration.test.ts`, describe `a turn restored mid-turn on a remote Runner` (real PostgreSQL): | Test | On master | Here | |---|---|---| | `closes the turn as a lower bound, also after the controller restarts between restore and terminal`: the restore is journaled by one receiver and the terminal arrives at a rebuilt one; replaying everything again is idempotent | fails | passes | | `keeps the turn's own final report when it arrives after the restore` (guard) | passes | passes | | `keeps a turn started after the restore pending, and journals nothing for a restore with no open turn` (guard) | passes | passes | - `native-session-executor.test.ts > binds retained warm transport usage to the successor run authority` covers routing through the session-authority filter: - a current-run restore reaches the successor's capture; - a draining-run restore reaches the drained capture; - an unrelated-run restore resolves with no capture; - recovery routing ignores a source restore. It fails when the filter is reverted to `isNativeUsageAccountingEvent`. - Related suites pass: `native-usage-delivery.integration` and `native-usage-accounting` (46 tests), and the usage, accounting, receipt and authority cases in `native-session-executor.test.ts` (224 tests). Server typecheck passes. - **The live remote path is unproven.** No Daytona wave ran because the Daytona Runner image has not been rebuilt yet. The local Runner counterpart (#20517) passed restart-chaos L1, seed 20386, 4/4 (`w-20261110T104332-s20386`). ## Risks - **Upgrade safety.** Usage state saved by current master has no journaled restore and rebuilds as before. A receiver that predates this change replays a journaled restore as an ignored event, which is the behaviour on master. A restore that an older controller already acknowledged is not replayed, so that run waits for the closure timeout as it does today. - **One extra receipt per mid-turn restore**, only for ACPX and OpenCode and only while a turn is open. Nothing is added before `turn.start`: a restore with no open turn returns immediately. - **Codex and other providers are unaffected.** They do not use turn accounting. - No schema or persisted-format change. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through Claude Code with shell, file-edit and code-execution tools. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
37808f2a4e |
fix(accounting): close a local-Runner turn completed across a session restore as a lower bound (#20517)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runs record usage receipts, and accounting settles a run's charge once its final receipt is complete. > - ACPX Claude reports provisional zero usage until a turn ends, then one final report. > - A restart can stop the provider after the agent's finish call and before that final report. The restored Runner then resumes the session and reports the same turn completed with no complete report. > - The receipt stayed incomplete, so the accepted run waited for the 10-minute closure timeout and then settled as unpriced. Its spend was missing from company totals in the meantime. > - This pull request closes such a turn as a lower bound, the same way a turn the Runner fails as lost on restore is already closed. > - The benefit is that an accepted run settles right away with its observed usage and an honest lower-bound label. ## Linked Issues or Issue Description Refs #20484 (restart-chaos L1 contention follow-up). **What happened?** After #20484 fixed the database contention, restart-chaos L1 (`--seed 20386 --db-pressure 4`, codex and claude, K1) reached 4/4 tasks done. Each task still failed the T-oracle `company_spend_settled` with `pending_runs=1`. A replay on master with `--keep` (`w-20261110T100747-s20386`) reproduced it, and I read the kept database: - The pending run (`71f71d10`, ACPX Claude) succeeded and had its result accepted. Its usage had `accountingReceiptReady: false`, zero tokens, `costStatus: unpriced`, and no cost event. - Its only turn started before K1, and every usage report in it was provisional (`runDeltaComplete: false`). K1 struck right after the agent's `paperclip_finish` call (source seq 118), before the turn's final report. - On restart, recovery verified the local process stopped and resumed the session (`session.resumed`, seq 119). The restored Runner then reported the same turn `turn.completed` (seq 124) with no usage report. - The other Claude run in that wave lost its turn as `provider_turn_lost_on_restore`, ran a new turn live with full usage, and settled at once. **Expected behavior** An accepted run whose open turn's final report was lost with the provider process settles promptly. It records its observed usage as a lower bound, like a turn lost on restore. A complete report that does arrive after the restore is still used. **Steps to reproduce** `server/src/services/native-runtime/native-usage-accounting.test.ts > closes a turn that completed across a session restore without its final report, as a lower bound` uses the event sequence from the kept database. It fails on master because the snapshot stays incomplete. ## What Changed - `createNativeTurnAccounting` observes `session.resumed` and marks every open turn as restored. A terminal event for a restored turn without a complete report marks it lost. The snapshot then closes it as a lower bound (`accountingUsageLost`), through the existing lost-on-restore path. A new `turn.started` clears the mark. - A lost terminal turn keeps its identity in the snapshot, so the executor's terminal-turn match certifies the final checkpoint as complete. This came from review. - The native session executor includes `session.resumed` when it rebuilds turn state from the run's event log after a controller restart. - `doc/SPEC-implementation.md` documents the case next to the existing lost-on-restore rule. ## Verification - Red on master, green here: `native-usage-accounting.test.ts > closes a turn that completed across a session restore without its final report, as a lower bound`. It also checks that replaying the history twice gives the same snapshot. - Red on the first revision, green after the review fix: `native-session-executor.test.ts > certifies a turn completed across a session restore as a lower bound (history replay: false|true)`. It drives the restored sequence through executor completion, both live and from the event history. - New guards that pass before and after: - `keeps a restored turn's own final report when it arrives after the restore`: a complete report after resume is used, not marked lost. - `keeps an incomplete turn pending when the session resumed before it started`. - The existing `keeps an incomplete provider report pending instead of trusting placeholder zeroes` still passes. - Related suites pass: `native-usage-accounting`, `native-usage-delivery.integration`, `closed-native-receipt-accounting` (107 tests), and the usage, accounting and receipt cases in `native-session-executor.test.ts` (151 tests). Server typecheck passes. - Live, restart-chaos L1, seed 20386, `--db-pressure 4`, codex and claude, K1, head `a6e521f06c`, clean worktree: `w-20261110T104332-s20386` passed 4/4. R1–R6 and the T-oracle passed for every task. `company_spend_settled` reported `pending_runs=0` (summary 85.30823 = cost events 85.30823), with zero serialization failures under about 397k pressure updates. On master the same seed failed every task on `company_spend_settled pending_runs=1` (`w-20261110T100747-s20386`, among others). ## Risks - A restored turn without a complete report now settles at its observed usage, flagged as a lower bound, instead of after 10 minutes as "closure timed out". In both cases the final report never arrived. The money recorded is the same observed usage; only the timing and the provenance label differ. - Scope: local Runner only. The executor fallback-recorder path sees `session.resumed`. The producer-owned delivery path for remote runners (`native-usage-delivery.ts`, `isNativeUsageAccountingEvent`) still drops it, so a remote ACPX/OpenCode turn restored mid-turn still waits for the closure timeout. That is a follow-up. - No schema or persisted-format change. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through Claude Code with shell, file-edit and code-execution tools. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
0e325e8d52 |
fix(runner): settle saved handoffs after verified shutdown (#20336)
## Thinking Path > - Paperclip manages AI agents and their work. > - An agent can save a question or approval handoff before its provider stops. > - A missing provider terminal notification does not prove that the saved work failed. > - Runner already verifies provider exit, event drain, tool settlement and suspension. > - This change uses that proof to settle the same saved handoff. > - Unproven shutdown holds continuation and explains that the work is saved. > - Recovery must never repeat the completed work or allow overlapping execution. ## Linked Issues or Issue Description Refs #20330. That PR extends the Codex terminal drain window. This change handles the separate case where no terminal arrives but the owned shutdown can prove stopped execution. **What happened?** A native run saved a human handoff, accepted cancellation, and later stopped its provider. The missing terminal notification still made the attempt fail and show “Couldn’t finish this attempt.” **Expected behavior** Keep the saved handoff. Accept its yielded result after either an authoritative provider terminal or verified shutdown of the exact execution. If shutdown remains uncertain, retain the handoff and explain the hold without offering a duplicate Retry. **Steps to reproduce** Save a governed wait, acknowledge interruption, and omit the provider terminal. The new regression uses the real Runner binary and a controlled Codex process. The old runtime rejects the handoff instead of asking the existing shutdown owner to settle it. **Deployment mode** Native Runner with Codex. Local verification used macOS arm64 and Node 26.4.0. Hosted sandbox recovery has not been qualified. ## What Changed - Add an optional driver shutdown-settlement operation. Runner returns proof only after its existing stop, drain, tool settlement and suspension barrier passes. - Bind the receipt to the run, normalized session and turn. Persist it with the saved wait and validate it during recovery. - Complete the same yielded handoff after verified shutdown. Retire the stopped session from warm reuse. Do not invent a provider terminal or complete usage. - Retain a named shutdown-pending condition under the existing bounded same-run recovery policy. Preserve actual provider failures and integrity failures. - Show saved-work shutdown guidance in task and inbox presentation. Keep genuine approvals authoritative and disable duplicate Retry. - Document the completion and restart contract. The checkpoint field is additive; older checkpoints retain the terminal-or-shutdown path. ## Verification - Red: the new runtime regression failed against the original implementation because verified shutdown was never requested. Server presentation and error-classification regressions also failed before the fix. - Red: both inbox shutdown states showed generic text before the inbox change. - Green: Runner runtime, shutdown and harness-backend suites pass 301 tests. Tests cover missing terminals, failed stop proof, wrong identity, cancellation, late proof, genuine provider failure, and restart after receipt persistence. Recovery completes once without another model turn. - Green: the native executor and task-presentation suites pass 1,371 tests. Execution projection, inbox and blocker presentation pass 125 tests. Two test-authoring mistakes were corrected: an integrity-error text expectation and an assertion for a nonexistent projection field. Their initial failures remain in local logs. - Runner TypeScript and full server layered type checks pass. Token gates and diff whitespace checks pass. - Live provider acceptance: real Runner and Codex with gpt-6.1-sol, one disposable local workspace, one tool call, and one deliberately dropped terminal notification. The saved file, bound settlement receipt, one accepted yielded result, one provider turn, confirmed provider exit and suspended Runner all passed. The tool handler and persistence port were controlled. This does not claim a full server or hosted journey. - The full browser question-and-answer cell stopped before execution because OPENAI_API_KEY was absent. No model work ran in that attempt. Hosted sandbox recovery and the full browser continuation journey remain unverified. - `pnpm verify` returned exit 2 because the execution-spec change requires broad CI. Required CI will supply that coverage; a full local repository suite was not repeated. - Review correction in `8553da244e`: a failed trailing-event write could hide a provider failure. A new regression reproduced red, then passed after all ordinary drain errors retained the named pending condition. No settlement receipt or result can be accepted after that error. The live missing-terminal check passed again on the corrected runtime. - The real PostgreSQL control-plane regression and three existing governed-wait controls pass. The new case persists one accepted yielded result, a bound shutdown receipt, and the original pending human question without a provider terminal. - First CI revision `7c2cb9b1`: test-program typecheck found a missing `PrpEvent` annotation in the new fixture. The annotation is fixed and the Runner test-program typecheck passes. Runner support also timed out in the unchanged `registered roots remain protected even when equal to genuine public static bytes` case after 30 seconds. One unchanged local diagnostic passed in 3.69 seconds overall. The hosted timeout cause remains unknown; no timeout, fixture, or assertion was weakened. [Original failed support job](https://github.com/paperclipai/paperclip/actions/runs/42670851821/job/129346900133). - Rebased onto the merged #20330 and current recovery changes. The combined branch passes 1,402 executor/presentation tests. Both prompt and 18-second provider terminals still pass. A warm session with no terminal passes after the full 30-second drain and verified stop/suspension. Its initial overly broad assertion rejected a nonterminal turn event; the corrected assertion rejects every terminal type. The rebased CI gate was cancelled before any runner or step started; its cause is unconfirmed and it supplies no validation credit. Final revision `b49ec5a144` starts fresh CI and review. - Final-head CI attempt 1 failed before creating jobs. GitHub reports an internal server error (correlation `2c576b07-b67f-4179-a69f-74f7d99cd926`), followed by empty runner-selection outputs for the aggregate jobs. One unchanged-commit retry was requested; the initial admission failure remains recorded. [CI run](https://github.com/paperclipai/paperclip/actions/runs/42673159868). - **Current blocker:** final head `b49ec5a144e94e41e96e2508a79bd9aec91fe45a` has a fresh 5/5 Greptile review and zero unresolved threads, but required CI is not complete. GitHub reports an active [Actions and Checks API incident](https://www.githubstatus.com/incidents/tp8cghg7tlc4). Attempt 2 ran only part of the full selected graph. Build, canary, Rust, sandbox providers, admitted browser shards and other completed test jobs passed. The E2E aggregate correctly failed because `policy` and `e2e_shards` had `abandoned` results. Server shard 5/12 received a runner shutdown signal at 18:36:55 UTC and its test step was cancelled; its job still reports in progress. The shutdown cause is not established. The separate review workflow had the same pre-execution internal-error class and passed one bounded retry. No further retry was submitted during the incident. Resume with the missing/cancelled coverage after GitHub recovers; do not count the absent typecheck, Runner or support jobs as passes. [Failed coverage gate](https://github.com/paperclipai/paperclip/actions/runs/42673159868/job/129354318236), [interrupted server shard](https://github.com/paperclipai/paperclip/actions/runs/42673159868/job/129350854174). ## Risks - A missing terminal can use one additional bounded shutdown interval. Unknown shutdown still blocks continuation. - Provider terminal and usage accounting keep their separate meaning. The receipt proves stopped execution and settled tool outcomes, not model success. - Recovery rejects a foreign receipt, a later turn, an active turn with a settled receipt, and an actual failed provider terminal. - There is no database migration. Older checkpoint shapes remain readable. Drivers without this optional proof retain a named pending state. - This PR does not deploy the fix or repair an existing hosted task. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, repository tools and local code execution. The exact serving model ID and context window are not exposed in this session. Live test inference used gpt-6.1-sol. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Dotta
|
a3e7e63330 |
Guard Native recovery with process-space proof and explicit containment (#19813)
## Thinking Path > - Paperclip manages agents and their work. > - Recovery must preserve execution ownership and accepted results. > - A saved PID proves nothing outside its recorded process namespace. > - Uncertain unfinished work must stay held until its original owners stop. > - Operators need a source-bound containment action and a separate ordinary Retry. > - Accepted results must use the existing finalizer without another provider turn. > - This PR keeps those paths safe, recoverable, and quiet between real obligations. ## Linked Issues or Issue Description Refs: #19713 Refs: #19591 Recovery could use a local PID to release or replace a controller from another process namespace. Connected-chat Retry also rejected successful containment on a task in `todo`. The new containment action used a broader permission than the existing owner-stop action. A later review found that result-less guards also rejected the existing accepted-result confirmation action. ## What Changed - Record and verify the controller's kernel/PID namespace. Keep unknown and foreign unfinished ownership held. Preserve independent durable stop evidence and the existing remote lease fences. - Use the shared accepted-result ownership reconciler. Bind settlement to the observed lease token, generation, boot, PID, namespace and process birth. Retain finalizer, cleanup, current-controller, live-owner and remote exclusions. - Share the remote-owner reader across recovery consumers. Scope runner leases to the same company, task and run. Do not make local PID probes for remote accepted-result ownership. - Gate original-host containment on `runtime:confirm_owner_stop`. Bind it to the current task, source, assignee, original expiry, lease and Stop state. Audit separate controller/provider containment and reviewed action outcomes. - Keep containment separate from ordinary Retry. Use one canonical committed receipt and the normal admission, budget, workspace and lineage checks. Preserve the recorded external-action outcomes. - Keep the controller-only confirmation actionable for both result-less work and accepted results. Result-less work gets its next source-bound attention in the same transaction. Accepted results keep their original result and use normal shared settlement/finalization. - Keep identity inspection quiet. Preserve the original foreign-owner expiry and use the existing scoped finalizer in accepted-result recovery controls. - Update the recovery card, validators, execution projection, docs and access maps. Reuse existing canonical columns; add no migration or ownership backfill. ## Verification Current publication head: `71ea76b568de09d6313d8992e803131d9d5c54f3`. **The current fix has a causal red/green check.** On the test-only revision, both foreign-local stopped/fenced controller cases failed because recovery probed saved runner, process-group and provider PIDs in this host's namespace. This measured the wrong probes; it did not run duplicate providers. After the fix, the same two cases passed every no-probe, no-claim, no-controller-stamp, no-stop-receipt and unchanged-authority assertion. Three existing remote preparation, grace and fenced-controller progress cases also passed. Total: **5 passed, 0 failed;170 filtered rows excluded**. All three normal affected App TypeScript programs passed on this head (660,681 and428 roots). The fix separates a controller wait from execution stop proof. Independent live/grace evidence still schedules recovery. A stopped/fenced foreign controller can pass this namespace guard only for remote execution, which keeps its later checkpoint and never-launched checks. Foreign local execution stays held before runner or provider PID inspection. Earlier source-scoped results remain evidence, not fresh executions on the current head: - On `c86dd741`, the complete failed-owner file passed **25/25**, with no skips or filters. Empty scans wrote no controller stamps. Foreign ownership kept its original expiry. Normal returned-claim settlement and attention at the exact deadline passed. - On `c86dd741`, **15 selected accepted-result/restart controls passed**. The81 filtered rows are excluded. Both completion cases used the existing scoped finalizer and kept their accepted result without another provider turn. - Trusted CI at synthetic merge `50d072a5` (actual master `1b7fe334` plus `c86dd741`) passed the complete **17-case maintenance-server file**. This includes the original90-real-second quiet observation and two qualified hourly windows. The quiet case took118,918ms. The four jobs that failed on the earlier publication all passed on that synthetic merge. - On `6243b32c`, the complete ownership-report file passed **29/29**. Native NULL namespace stayed unprovable and generic compatibility remained. Three selected recovery controls passed;170 filtered rows are excluded. - On `f81f2c3c`,356 business cases passed: whole ownership-route221, recoveryUI70, owner-proof38 and controller-confirmation8; selected accepted-result authority18; and one real connected-chat containment, separate authorized Retry, useful Native result and downstream reply case. The2,024 filtered rows are excluded. A separate earlier accepted-result causal case kept the original result and completed after restart without replay. Normal Shared exports, App programs, UI typecheck, token gates and map checks passed on their recorded revisions. Unchanged DB, adapter, Eval and Runner inputs/outputs have explicit carry proofs. The clean normal merge with fetched master `8fe524f7` preserves all31 exclusive owned paths and14,827 unrelated master entries. The nine overlapping paths were reviewed, including the upstream unidentified-run stop action. Fresh CI must validate that actual merge composition. PG cases used existing external PG17, fresh owned databases, pools at most4, one worker and no SQL observer during cases. Registered clients and recorded process groups closed. Independent final SQL audits found zero campaign sessions before normal database cleanup. Original failing fixtures remain retained. Original failures are preserved. One new test first failed compilation due to a missing getter import; an import-only correction passed its full owning programs before the causal red/green check. A local quiet attempt stopped in the normal prerequisite hook before any business case and its collector rejected two normal build markers. The later17-case CI pass establishes quiet recovery; it does not turn that failed local collector into a pass. An earlier useful case passed before its private cleanup collector failed on an owned read-only directory; normal cleanup completed after all owners closed. Earlier runner-shutdown cancellation and a separate private whole-suite failure remain unexplained. Current-head checks are complete:57 success,1 neutral and3 skipped, including the successful aggregate CI gate. The legacy status passed. Greptile is5/5 and Ponytail gives an explicit5/5 Ship verdict for this head. All seven review threads are resolved. Full recursive typecheck, the full test suite and release build were not repeated locally. No result here proves deployed suspension, a live provider or termination on another host. ## Risks - Unknown physical ownership can require an owner/admin action. A false containment statement can be unsafe. - New work, reassignment, Stop, renewed authority, accepted results or pending cleanup can invalidate an action. The operator must refresh. - Controller confirmation does not prove provider/tool containment. Completed external actions must not be replayed. - Generic legacy and child-process compatibility remains separate from the strict Native controller policy. - Existing saved state and canonical schema are preserved. Permanent safety refusals remain held with their normal named recovery path. ## Model Used OpenAI `gpt-6.1-sol` agents implemented and tested the changes with reasoning, repository tools and code execution. A GPT-6-based Codex managing agent reviewed and integrated them. The host does not expose the manager's exact build or the context-window sizes. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs: #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge |
||
|
DottaandClaude Opus 5.5
|
2dd7a98753 |
fix(recovery): return a settled restore hold's warm sandbox to reuse (#20278)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Recovery tells a waiting parent or dependent when a child or blocker stalls on the board (`issue_blocker_stalled`) > - A legacy child that an older release bound to its parent's remote workspace failed its save, and its save recovery was exhausted. Its ended run still holds the shared workspace, days past the one-hour holder cutoff > - Workspace reservation refuses a run on such a hold and fails setup with "The workspace still has an unfinished writer or save operation." Every stall wake of the parent created a run that could only fail this way > - The stall sweep treated each failed run as an undelivered wake and retried it every ten minutes, with no backoff and no limit. On staging this made 72 failed runs (58 of 60 failures since a deploy), in pairs every ten minutes > - This pull request makes the stall sweep hold a wake while the waiter's workspace hold refuses a run, and bounds retries of a wake whose run fails > - The benefit is that a held workspace no longer produces a failed run every ten minutes, and the wake still arrives once the hold settles ## Linked Issues or Issue Description **What happened?** A parent task was woken by `issue_blocker_stalled` twice every ten minutes. Each run failed in setup (`setup_failed`, `executionRecovery.kind=bootstrap`, `providerWorkStarted=false`) with "The workspace still has an unfinished writer or save operation." The parent's remote workspace was held by a child's ended writer run whose save recovery was exhausted (`native_workspace_sync_out_retry_exhausted`, last failure `agent_stop_unconfirmed`). The child shared the parent's workspace because an older release created it before subtasks got their own workspace (#19860). **Expected behavior** A wake that cannot start because its workspace is held must not create a failed run every time. It should wait until the hold settles. Automatic retries must be bounded. **Root cause** - `server/src/services/heartbeat/workspace-preparation.ts` rethrows `RemoteWorkspaceWriterActiveError` as a setup failure when the ended writer is silent past `WORKSPACE_BUSY_HOLDER_STALE_AFTER_MS` (documented, intended). - `server/src/services/recovery/blocker-stall-wakeups.ts` retried every non-delivered wake row (including `failed`) after `REFUSED_WAKE_RETRY_MS` (10 minutes) with no backoff and no limit, and it did not check whether the waiter could start at all. ## What Changed - `remote-workspace-writer-hold.ts`: add `remoteWorkspaceHoldRefusesStart`, the one rule for a hold that reservation refuses outright (unattributed writer, or a writer silent past the cutoff with no live owner fence). - `workspace-preparation.ts`: use that rule instead of an inline copy, so reservation and wake senders cannot drift. - `blocker-stall-wakeups.ts`: before waking a waiter, read its bound remote workspace hold. When the hold refuses a run, do not wake it; count it as deferred and re-check at most hourly. A workspace with an unchanged-release baseline is left to reservation, which can release it. - `blocker-stall-wakeups.ts`: a stall wake whose run failed is retried after 10, then 20 minutes, and not after the third failed run. Refusals (wake-on-demand off, agent paused) keep the 10-minute retry. A new stall (new recovery action) wakes again. - `doc/execution-semantics.md`: document both rules. ## Verification - Red before the fix (`server/src/__tests__/issue-blocker-stall-wakeups.test.ts`): 3 new tests failed. The held-workspace test got a delivered wake; the backoff test got a third wake 11 minutes after the second failure. - Green after the fix: `npx vitest run src/__tests__/issue-blocker-stall-wakeups.test.ts` passes 20/20. - The new tests use the staging shape: a legacy `shared_workspace` remote workspace owned by the parent, a child bound to it, the child's failed native writer run ended four days ago with `native_workspace_sync_out_retry_exhausted`, a dirty authority, and the child's board-owned exhausted recovery action. They also prove that a writer inside the cutoff still gets its wake (the run parks in a workspace wait), and that the wake is delivered once the hold settles. - Related suites pass: `heartbeat-remote-workspace-busy.test.ts`, `heartbeat-dependency-scheduling.test.ts`, `native-cancelled-workspace-preservation.test.ts`, `src/modules/run-dispatch`. - `npx tsc --noEmit -p server` passes. - `pnpm verify` exits 2 because `doc/execution-semantics.md` is a global CI input; required CI supplies the broad coverage. ## Risks - Low. No schema or persisted-shape change. Saved state from the previous release (existing wake rows, existing workspace authority) is read as-is: earlier `failed` stall wake rows count toward the new three-run limit, which stops the current loop at once after deploy. - A parent whose workspace stays held is not woken for the stall until the hold settles. The stall is still listed on its page and the child's recovery action stays with the board. - Not fixed here: the exhausted save hold itself is never settled automatically (`agent_stop_unconfirmed` exhaustion in workspace recovery), and legacy children that still share a parent's workspace are not moved. Both touch workspace recovery code that is under concurrent change and are left for a follow-up. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, browser read-only inspection of staging). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
6459488984 |
fix(native): open a remote replacement as a new runner after archiving the reattached one (#20242)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - After a restart, a remote (Daytona) run can resume by reattaching to
its surviving sandbox runner. If provider recovery fails and the
checkpoint allows it, the run opens a replacement provider session
instead.
> - Before a replacement, the executor archives the local controller
state and the remote session root as a continuity break, then opens a
new session.
> - On a reattach backend, the replacement still adopted the archived
runner and asked for its old authority, so it failed. The archive had
already moved the remote state, so the next retry could not verify the
runner and the issue waited 17 minutes on an unconfirmed stop.
> - Review showed that a reattach backend fixes many launch inputs once
at creation (runner binary, provider pack and companions, artifacts,
live-recovery mode, authority, spawn barrier), so resetting them after
the archive does not scale.
> - This pull request builds the replacement on a new backend for the
same run, created under a `bootstrap_incomplete` claim after the
archive, so every launch input resolves through the ordinary fresh path.
> - The benefit is that a failed reattach after a restart falls back to
a working replacement instead of a stuck issue.
## Linked Issues or Issue Description
**What happened?**
In restart-chaos Daytona K13 storm wave seed 9503, task t36 graded R1
(no rest within budget). A K2 drain suspended run a53dd948's sandbox
runner (pid 589) during startup. On restart the run claimed
`reattach_remote_runner`. The adopted runner authenticated and restored,
then its connection closed, and provider recovery failed. The runtime
opened a replacement: `provider.session.archive_before_replacement`
archived the local and remote state, then the replacement failed with
"PRP provider resume state is unavailable". The same-run retry at
10:25:50 then failed with `runner_remote_state_unavailable: exit=1` (the
remote `runner-state.json` had moved), and file preservation deferred on
`agent_stop_unconfirmed` until the wave ended.
**Expected behavior**
After the continuity archive, the replacement opens a new runner for
this run, as it does for every other claim kind.
**Steps to reproduce**
1. Run a remote (Daytona) native run and restart the server so the run
claims `reattach_remote_runner`.
2. Make provider recovery fail on the adopted runner with a checkpoint
that allows replacement.
3. See the replacement open on the adopted runner's authority and fail
after the archive.
**Paperclip version or commit**
master at
|
||
|
DottaandClaude Opus 5.5
|
3374ca54f5 |
fix(runner): record a runner exit mid-turn as runner loss, and stop the lineage sampler panicking (#20259)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The native runner (`runnerd`) runs each provider turn, and the server records why a run stopped and what the person can do next > - When `runnerd` exits during a provider turn, the server saved the run as `native_provider_terminal_failed`: "The provider session is permanently unusable", with automatic retries stopped > - The real cause was the runner process, and the transport already names it (`native_runner_process_exited`) at the start of the closed-stream message > - Separately, the process-lineage sampler added in #19727 panicked on Linux debug builds when a provider tool started its own session, and that panic text became the run's visible failure > - This pull request records a runner exit mid-turn as runner loss, and reads session IDs without the assertion that panicked > - The benefit is that a person sees the true cause of a stopped run, and recovery treats a lost runner like every other runner exit ## Linked Issues or Issue Description Refs #19727 **What happened** A chat eval killed only the owned `runnerd` process (SIGKILL) while a provider tool was running. The run failed with `errorCode: native_provider_terminal_failed` and the next action "The provider session is permanently unusable". The error text was `Notification transport failed closed: native_runner_process_exited: runnerd exited unexpectedly: paperclip-runnerd panic: panicked at rustix-1.1.4/src/pid.rs:61:9: assertion failed: raw > 0`. The panic came from the lineage sampler thread before the kill. It called `rustix::process::getsid`, which asserts a nonzero session. A kernel thread, or a process whose session leader is outside the PID namespace, reports session 0. **Expected behavior** A runner process exit is recorded as `native_runner_process_exited`, the same cause the server already uses for a runner exit outside a turn. Recovery then applies its normal runner-loss rules (checkpoint evidence, bounded attempts, cleanup quarantine). The lineage sampler does not panic, and its diagnostics do not replace the real failure text. **Steps to reproduce** 1. Start a native OpenCode run on Linux with a debug `runnerd`. 2. Let the agent run a shell tool that starts its own session (for example a long `node -e` command). 3. Send SIGKILL to the `runnerd` PID only. 4. Read `GET /api/heartbeat-runs/<runId>`: `errorCode` is `native_provider_terminal_failed` and the error names the rustix panic. ## What Changed - `nativeSessionFailureSourceCode` returns `native_runner_process_exited` for a `NativeProviderTerminalFailure` with code `notification_transport_failed` when its message starts with `Notification transport failed closed: native_runner_process_exited: `. Only the runner transport writes that prefix. Other closed-stream causes stay `native_provider_terminal_failed`. - `paperclip-process-table` has a new `session(pid)` function. Linux reads the session field of `/proc/<pid>/stat`. macOS calls `getsid`. A session of 0 or an error gives `None`. - `process_lineage` reads sessions through `paperclip_process_table::session` instead of rustix `getsid`. - Regression tests: the runner-loss classification (attempts 1 and 3, and three negative shapes), the stat session parser with a kernel-thread line, and a live read of this process's session. ## Verification - `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts -t "runner process exit that closed|preserves stable provider and runner|model rejection from a provider terminal|operator-owned through terminal"` (8 passed) - `cd packages/paperclip-runner/runner && cargo test -p paperclip-process-table` (macOS: 3 passed; the Linux-only parser tests run in CI) - `cd packages/paperclip-runner/runner && cargo test -p paperclip-runner-core --lib process_lineage` (6 passed) - End to end: the `recovery.*` chat eval cells that kill `runnerd` are run against this change on a separate eval branch. ## Risks - Behavior shift: a runner exit during a turn now follows runner-loss recovery instead of stopping as a permanent provider failure. Recovery still decides from durable evidence. Ambiguous state stays terminal, and an unverified cleanup stays quarantined. - Previous-release story: no persisted shape changes. Runs saved earlier with `native_provider_terminal_failed` keep that code. Only new failures are classified with the new rule. - The session read on Linux now uses `/proc` instead of a syscall. It gives the same value, and it reports 0 (now `None`) in the same cases. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
3f2a940d5b |
fix(runner): answer held Paperclip tool calls after a local restart before the next turn (#20227)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a restart interrupts a turn, the Runner keeps each Paperclip tool call the model made as "held" until Paperclip answers it, and it refuses to start the next turn while any call is held > - Paperclip answers held calls after a reattach through its tool recovery path. That path was wired only for remote runners > - In restart chaos wave `w-20261109T084023-s9403`, t101 (local codex, K2 then K1) held a `read_document` call that nothing answered, so the continuation's `turn.start` failed and the task needed a Retry > - This pull request wires held-call recovery for local runners too, and makes `turn.start` answer or wait for held calls first > - The benefit is that a turn lost to a restart no longer strands the run behind one unanswered call ## Linked Issues or Issue Description Refs #19831. Restart chaos wave `w-20261109T084023-s9403` (master `fc5c79a327`), task t101 (codex C1, K2, local). **What happened:** - Run `4b5810d5` called `read_document` (`exec-ac8fa9b4`) around the K2 drain at 09:08:18. A K1 then killed the next server start. - At 09:08:36 the third controller reattached. The Runner restored the conversation and reported the old turn lost (`provider_turn_lost_on_restore`, recoverable). The held call stayed pending, by design: a held call accepts a late result. - The controller had the call: the quarantine detail lists `exec-ac8fa9b4` under `semanticTools.pending`. But `recoverPendingSemanticInputs()` returned at once: for a local run `controllerPersistence` is unset, so `recoverSemanticToolInput` was `undefined`. - At 09:08:46 the continuation's `turn.start` failed with "cannot prepare the next provider turn while tool calls are active (pending: 1 [read_document:exec-ac8fa9b4…])". The run ended `native_session_interrupted`. ## What Changed - `server/src/services/native-runtime/native-session-executor.ts`: the runnerd transport now always gets `recoverSemanticToolInput`, not only when `controllerPersistence` (remote) is set. It uses the same epoch check and `SessionAuthorityEpoch.recover`. Recovery re-authorizes and re-runs reads, and replays receipts for mutations; it never repeats an uncertain mutation. - `packages/paperclip-runner/src/live/runnerd-codex-transport.ts` (`#startTurnWork`): when the control plane has unsettled semantic inputs, it runs `recoverPendingSemanticInputs()` before queueing `turn.start`, and waits for that pass, bounded by the turn-start deadline. Recovered results are queued as commands ahead of `turn.start`, and the Runner applies commands in order. If recovery has not finished by the deadline, the turn start fails with `native_turn_start_held_tool_calls_unresolved` instead of sending `turn.start` behind it. When everything is settled there is no wait, so a normal turn start is unchanged. Live calls of a turn being replaced are not waited on; the replaced start still rejects them. - `packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts`: `recoverPendingSemanticInputs()` passes are serialized. A call made while the startup pass is still answering a held call waits for that pass, then runs its own, so the turn-start gate cannot finish early by skipping a call that is already being recovered. ## Verification Red/green: - `native-session-executor.test.ts` "answers a local run's held Paperclip tool calls after a restart (restart chaos t101)". On master `recoverSemanticToolInput` is `undefined` for a local run; with this PR it is wired. - `runnerd-codex-transport.test.ts` "answers a held call before starting the next turn (restart chaos t101)", against real runnerd and fake Codex, with a slow startup recovery pass already in flight when `turn/start` arrives. On master `turn/start` never ran recovery, and without the serialization `turn.start` is queued before the held call's result. With this PR the held call is recovered once and its result is queued before `turn.start`. Regression and static checks: - `runnerd-codex-transport.test.ts`, `durable-prp-control-plane.test.ts`, `runnerd-codex-transport-adoption.test.ts`: 510 passed. - `native-session-executor.test.ts` and `native-session-resume.test.ts`: 1368 passed, 1 failed. The failing case, "archives a damaged prior epoch … with real runnerd", fails identically on master on this Mac (`ENOENT … authority-epochs`), so CI is its reference. - `tsc --noEmit` passes for `packages/paperclip-runner` and `server`. ## Risks - Local runs now recover held calls after a reattach, as remote runs already did. Reads re-run under current authority. A mutation without a saved receipt still returns no result and stays held, as before. - `turn.start` may now wait up to its existing deadline while a recovery pass runs, then fails clearly if it has not finished. It never waits when nothing is unsettled, so normal turns get no added latency before `turn.start`. - No schema change and no change to persisted run state formats. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
3964ee1fca |
fix(native): a governance pause Stop with no turn result stays a cancellation (#20186)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A budget hard stop pauses the company; the pause's lifecycle Stop cancels the agents' running native turns > - When a Codex turn is interrupted this way, it ends without a semantic result > - The native executor treated that missing result as a retryable interruption, so it rewrote the cancelled run to `failed` and scheduled an exact resume > - The pause held that recovery, and Raise budget and resume continues only runs that are still cancelled, so the task stayed blocked with no new run > - This pull request makes the executor treat an acknowledged governance pause Stop like a person's Stop: the run stays cancelled > - The benefit is that Raise budget and resume continues the task (#20144), with no manual Retry ## Linked Issues or Issue Description Refs #20144 (resume continues budget-stopped work), #20143 (governance pause stops are not failures), #20113 **What happened?** Restart-chaos wave w-20261109T051242-s9802 (H1, seed 9802), task t03 (Codex, fault K2). The company budget crossed its hard limit at 05:19:39.9. The pause's lifecycle Stop ("Agent lifecycle stop requested") cancelled run 59e976bc at 05:19:40.05, and Codex acknowledged the interrupt. The turn ended without a semantic result, and at 05:19:41.27 the executor rewrote the run to `failed` / `native_session_interrupted`, with an agent-owned `active_run_watchdog` recovery (exact checkpoint resume). The task went to `blocked`. After `raise_budget_and_resume` at 05:20:20, no run started for 14 minutes. **Expected behavior** A pause's Stop is intentional. The run stays cancelled with its pause record, and resuming continues the task once. **Steps to reproduce** Pause a company for budget while a Codex native turn runs, so the Stop interrupts the turn before it records a result. Before this change, the run becomes `failed` with a scheduled exact resume instead of staying `cancelled`. ## What Changed - `native-session-executor.ts`: the no-semantic-result branch now also accepts an acknowledged governance pause Stop (`isGovernancePauseStop` with an acknowledged native Stop receipt) as an intentional stop. It settles the coordinator as `native_retry_cancelled` and raises `NativeCancellationPendingRecoveryError`, as it already does for a person's Stop and for a reassignment. The run is not rewritten to `failed`, and no recovery action is created. - The Stop's cancellation record is written by its fence before dispatch, so the check does not depend on whether the terminal write has landed. ## Verification - `native-session-executor.test.ts`: new cases for "Agent lifecycle stop requested" with a `pauseStop`, and for "Cancelled due to budget pause". Each was red before the fix (it reached the retryable-failure write) and is green now. A control case shows that a system lifecycle Stop without `pauseStop` (termination) still takes the retry path. - Full `native-session-executor.test.ts`: 1309 passed. Server typecheck (`typecheck-layers`, low-memory): no errors. - Not run: the H1 chaos rerun. Phase 3 runs it after merge. ## Risks - A governance pause Stop that lands during a turn no longer schedules an exact resume of that turn. The task continues on resume through a new Retry (#20144) instead. - Termination and cleanup use the same lifecycle Stop reason. They are excluded because only the pausing path records `pauseStop`. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test execution). Extended reasoning enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
fa7950121d |
fix(native): move an interrupted bootstrap's session aside on the run's own created sandbox lease (#20154)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When restart recovery proves a remote bootstrap reached no provider work, it bootstraps the same run again in the same sandbox. Before relaunching, it stops the interrupted runner (#20103) > - The relaunch starts a fresh controller record, whose command ids restart at `command_prp_00000001`. The sandbox still held the interrupted runner's journal, which had already recorded that command id with different data > - The relaunched runnerd rejected it ("commandId was reused with different command data") and exited while waiting for `run.prepare`. Every same-run retry collided the same way until the retry budget ran out and the task blocked > - The proven interrupted-bootstrap session move (#20103) ran only for a resumed lease. A same-run retry on a per-turn sandbox reuses the run's own created lease, so nothing moved the interrupted session aside > - This pull request applies that same proof-gated session move on the run's own created lease > - The benefit is that a same-run bootstrap retry starts its runner on a clean journal, and the old journal is kept as evidence ## Linked Issues or Issue Description Refs #20103 **What happened?** Daytona per-turn restart-chaos M1 rerun w-20261109T013733-s9501 (master |
||
|
DottaandClaude Opus 5.5
|
6451d2bfbf |
fix(native): verify a staged attachment restart on the claimed sandbox lease (#20136)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A follow-up codex run on a per-turn Daytona sandbox gets a fresh sandbox, with harness state restored from the previous turn's checkpoint, and attaches the previous runner session to the new run through a staged attachment > - A graceful restart during that start suspends the run. The next controller reattaches to the surviving remote runner, after proving it is on the sandbox lease the restart claim names > - The staged-attachment path required the lease to have been acquired as `resumed`. A per-turn follow-up's lease is `created`, so every such restart was refused with `native_runner_staged_attachment_recovery_unproven` > - The run then needed a Retry > - This pull request binds the staged restart to the claimed lease the same way the ordinary remote reattach does: same provider lease, same remote directory, and not a replacement sandbox > - The benefit is that a per-turn follow-up interrupted by a restart continues on its own ## Linked Issues or Issue Description Refs #20103 **What happened?** Daytona per-turn restart-chaos wave w-20261108T225520-s9501 (master |
||
|
DottaandClaude Opus 5.5
|
776f4cb602 |
fix(native): restart an interrupted remote bootstrap on a resumed reusable lease (t06) (#20103)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native run on a remote sandbox must survive a control-plane restart; a bootstrap interrupted before its first turn is restarted on the same run > - With a reusable (warm) sandbox lease, the restart always resumes the same sandbox. The interrupted runner's state is still on disk there: never suspended, and with no provider turn > - The resumed check accepted only a suspended checkpoint or no state at all, so it failed closed with `resumed_not_suspended`, and the task needed a Retry (restart chaos t06) > - The pre-phase had already proven no turn was sent and had stopped the stale runner, but that proof never reached the resumed check > - This pull request carries that proof forward, for this run only, and starts the bootstrap again on the resumed sandbox > - The benefit is that a warm task interrupted during startup continues on its own after the restart ## Linked Issues or Issue Description **What happened?** In restart-chaos wave w-20261108T192459-s9501 (Daytona warm, claude C1), run 9bf86b59's runner reached transport-ready at 19:37:42 UTC. The control plane was then down (another task's K4, then this task's K1). Restart recovery classified the run `bootstrap_incomplete` and restarted it on the same run at 19:41:51. The reusable lease resumed the same sandbox. `inspectRemoteHarnessState` found this run's runner state with `lifecycle` other than `suspended`, and the run failed at 19:41:59 with `runner_ingress_unavailable: runnerd process failed: runner_harness_state_mismatch: resumed_not_suspended`. The task needed a manual Retry. Per-turn and non-reusable leases do not resume, so they take another branch and were unaffected. **Expected behavior** A same-run restart of a bootstrap that provably sent no turn starts the bootstrap again on the resumed sandbox, with no Retry. It must never take over a live provider session or another run's warm session. **Steps to reproduce** 1. Run a task on a reusable (warm) Daytona lease. 2. Kill the control plane after the runner connected and before the first turn, so the run is `bootstrap_incomplete`. 3. Before this change, the restart fails with `resumed_not_suspended` on the resumed lease. **Paperclip version or commit** master at |
||
|
DottaandClaude Opus 5.5
|
d007afbf86 |
fix(native): let a follow-up wait out the previous turn's warm release instead of failing busy (#20068)
## Thinking Path - With a warm lifecycle, one session serves consecutive turns on a task. A follow-up comment wakes the next run, which claims the warm session's instruction copy and reuses the session. - #20048 makes warm the default on remote sandboxes. Its merge condition includes "no restart or follow-up ever leaves a task needing Retry". - In the #20048 Daytona chaos wave, task t12's follow-up run was created 43 ms after the previous run settled. The previous run was still releasing its warm session, so `claimWarmNativeInstructionCopy` saw it busy and failed the run with `native_session_supervisor_busy`. The task then needed a Retry. - The release takes seconds and always finishes, so a follow-up should wait for it rather than fail. A session busy because another run is preparing or executing a turn is a genuine conflict and must still fail at once, as existing tests require. ## Linked Issues or Issue Description Refs #20048 **What happened?** Wave w-20261108T142906-s9501, t12 (claude, warm): run 5b839da4 succeeded at 14:46:36.224. The follow-up run f2bf2f05 was created at 36.267 and failed immediately with `adapter_failed` / `native_session_supervisor_busy`, from `claimWarmNativeInstructionCopy` called by `instruction-working-copy.prepare`. Its automatic retry also failed, and only a user Retry completed the task. **Steps to reproduce** 1. Run a warm native session turn to completion. 2. While that run is still releasing its warm session (post-turn instruction check, release, hand-back), admit the next run on the same session. 3. The next run's instruction-copy claim throws `native_session_supervisor_busy`. **Expected behavior** The follow-up waits for the previous run's release (bounded) and reuses the session. A session held by a run that is still preparing or executing fails admission at once, as before. ## What Changed - `server/src/services/native-runtime/native-session-executor.ts`: - When the provider turn ends, the executor records the session scope as settling (`settlingNativeSessionScopes`). The outer `finally` clears the mark once the run has handed every scope back. - `claimWarmNativeInstructionCopy` waits, polling every 25 ms for at most 30 s (`warmSessionReleaseWait`), while the warm entry is busy and its scope is settling. It then re-reads the entry and runs the existing checks. If the entry is still busy at the deadline, or busy for any other reason, it throws `native_session_supervisor_busy` as before. - No wait happens on the normal path (entry idle), so turn admission adds no latency. - `server/src/services/native-runtime/native-session-executor.test.ts`: two cases. - A follow-up claimed while the previous turn's post-turn check is held: it waits, then succeeds once the release finishes. It fails on master with the wave's error. - The release outlasts the bound: it fails busy. **Previous-release state:** the change touches only in-memory admission state, so no saved run or session state changes. ## Verification - `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts src/services/heartbeat/instruction-working-copy.test.ts`: 1301/1301 pass. - With the wait removed (behaviour on master), the "release finishes" case fails with `native_session_supervisor_busy`. - The existing concurrency tests ("fences owned instruction preparation: concurrent", and the settled-release ownership tests) still fail admission at once. - Not yet covered: the Daytona warm wave rerun for #20048. ## Risks - **A wait that never ends.** The wait is bounded at 30 s and happens only while the owner is past its turn, which always reaches the outer `finally`. Every caller of the inner executor runs inside that `try`, and attachment staging keeps the same session scope. - **Hiding a real conflict.** Only a settling owner is waited for. A preparing or executing owner fails at once, as the existing tests assert. - **Scope.** I tried a similar wait at execution admission, but it conflicted with deliberate fences that reject a run during a settled release (the existing "releases settled codex only under its current owner" tests). It is not part of this PR. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`, Anthropic), 1M context window, tool use and code execution, via Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
b32ef9f111 |
perf(budgets): judge time to first visible agent activity; keep time to first text as a reported metric (#20077)
## Thinking Path - The native speed budgets (plan invariant 8) judge how fast an agent responds. They judged `task.time_to_first_token`: time to the first deliverable **text**. - In the clean local restart storm (w-20261108T153612-s9401), Claude breached its TTFT budget (cold p50/p95 17.4/34.7 s against 16.5/30). The per-run spans showed this was not a control-plane wait: in all 49 cold runs (and 76% of warm), Claude started a tool call before writing any text. Its first visible activity came at 7.4 s p50 from run creation; first text came at 19.6 s. Codex never calls a tool first. - So first text charges model tool work as latency for tool-first adapters. What the user experiences as "the agent responded" is the first visible activity: a text delta, a tool-call start, or a thinking indicator. - The coordinator decided: budgets judge time to first visible agent activity from run creation, with the budget numbers unchanged; first text stays a reported metric with no budget for tool-first adapters. ## Linked Issues or Issue Description **What happened?** Local storm s9401: the Claude TTFT budget breached on every aggregate, while the control-plane steps (queue 0.05 s, prepare 0.3 s, runner startup ~4 s, first provider event 1.3–1.8 s) were well within budget. The rest of the first-text wait was Claude streaming tool input (about 5.5 s per call) and running tools before it wrote text. **Steps to reproduce** 1. Run a Claude task whose first action is a tool call (most real tasks). 2. Read the run log: `provider.time_to_first_agent_event` ends at the tool call; `task.time_to_first_token` ends many seconds later at the first text. 3. The speed budget judges the later one and reports a breach. **Expected behavior** The budget judges the first visible agent activity from run creation (`task.time_to_first_agent_activity`). Time to first text is still recorded and reported. ## What Changed - `server/src/services/native-runtime/native-session-executor.ts`: when the first meaningful agent event commits (the existing `firstMeaningfulAgentEventKind`: text delta, tool-call start, or reasoning), record `task.time_to_first_agent_activity`. It uses the same detached anchor resolution as TTFT (accepted comment, answered question, durable wake request, or run creation), so it adds nothing to the turn path. - `server/src/services/native-runtime/native-speed-budgets.ts`: the metric is now named `time_to_first_activity` (was `ttft`), with identical budget numbers. The type comment documents the definition and why first text is unbudgeted. - `server/src/services/native-runtime/native-run-trace.ts`: the budget check judges `task.time_to_first_agent_activity` from run creation, keeping the existing input-wait handling. `task.time_to_first_token` is no longer budgeted. - `tests/restart-chaos/ttft-baseline-stats.ts` and `ttft-baseline.ts`: report first activity from run creation (the budgeted value) next to TTFT. - Docs: `doc/observability.md` (new span) and `doc/plans/2026-11-07-ttft-baseline.md` (metric table). - Tests: - A tool-first turn records first activity at the tool call (5 s) while TTFT records first text (16 s). - A tool-first trace with slow first text but fast first activity is within budget. - The existing budget tests now judge the activity span. Observability path (AGENTS.md section 5.7): the new run-log span uses only the existing `eventKind` and `anchor` attributes. Its name survives run-log redaction intact; checked with `redactEventPayload`. The operator OTLP gate is unchanged. It is not Telemetry. ## Verification - `cd server && npx vitest run src/services/native-runtime/native-speed-budgets.test.ts src/services/native-runtime/native-run-trace.test.ts`: 44/44 pass. - `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts -t "first visible activity|first assistant text once|first-token timing"`: 7/7 pass. - `npx vitest run --root tests/restart-chaos`: 159/159 pass; `tsc` on `tests/restart-chaos` is clean. ## Risks - **A metric rename.** Consumers of `run.performance.budget_exceeded` see `metric: "time_to_first_activity"` instead of `"ttft"`. In-repo there are none besides the tests; Phase 3's chaos speed grade switches to the new span. - **A looser budget for slow first text.** Intentional: first text is reported, not judged. A regression in control-plane latency still shows up in first activity. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`, Anthropic), 1M context window, tool use and code execution, via Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
c910651115 |
fix(native): start each rebound run with a fresh restart-continuation marker (#20053)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When the control plane restarts during an agent turn, the runner restores the same conversation. A Codex, Claude (ACPX) or OpenCode turn lost in the restart is marked `provider_turn_lost_on_restore`, and the run continues it once in the same conversation > - That one continuation is guarded by a disposition-only recovery marker in the session checkpoint. It is run state: it names a turn of the run that used it > - A follow-up run (a user comment on the same task) reuses the provider session by re-binding the previous run's checkpoint to the new run. Re-binding clears the run-scoped turn, result and request state, but it kept the marker > - The chaos K13 storm (wave w-20261108T134237-s9401) hit this three times (t24 and t45 on Claude, t77 on Codex). The task's first run had already used its restart continuation. The follow-up run inherited a spent marker naming a turn it never had, so when a restart lost its own turn, the driver kept the session closed. The run failed, and the user had to click Retry > - This pull request resets the marker at both re-bind sites, as for every other run-scoped field > - The benefit is that a restart during a follow-up turn continues on its own instead of putting a Retry click on the user ## Linked Issues or Issue Description Refs #19831 (durable ownership plan, restart continuation). Found by the restart chaos campaign. In all three failed follow-up runs: - After `resume_dead_runner`, the run emitted `turn.failed {code: provider_turn_lost_on_restore, recoverable: true}` for its own active turn. - The checkpoint still carried `dispositionOnlyRecoveryConsumed: true`, with `dispositionOnlyRecoveryTurnId` set to the previous run's continuation turn. ## What Changed - `native-session-resume.ts` (`rebindNativeSessionCheckpoint`) and `native-session-executor.ts` (warm checkpoint rebind): a rebound run starts with `dispositionOnlyRecoveryConsumed: false` and `dispositionOnlyRecoveryTurnId: null`. - Nothing changes inside a run: the driver's reopen rule (`mapTerminalTurn`) and `nativeRestartInterruptedTurnId` are untouched. The first commit changed the helper; review found the driver keeps its own copy of the rule, and the second commit reverts the helper in favour of this fix. ## Verification - `native-session-resume.test.ts` "retains provider identity but clears prior turn and event state" now seeds a spent marker on the previous run. It fails on `master` and passes here. - `native-session-executor.test.ts` "starts a new run's usage from the prior run's total when a warm checkpoint resumes" (3 cases) seeds the first run's checkpoint with a spent marker and asserts that the second run starts with it reset. It fails on `master` and passes here. - The other `native-session-resume.test.ts` tests pass. One test, "archives a damaged prior epoch before completing guarded same-task replacement with real runnerd", fails locally identically with and without this change (ENOENT under `authority-epochs`, from the real-runnerd environment). It's left to CI. - Next: a targeted chaos rerun (Codex and Claude C1 under repeated K2) on this branch. I'll add the result here. ## Risks - **Previous-release story (upgrade-safe run state):** no persisted shape changes. A run re-bound by this build gets a fresh marker. A run re-bound before the deploy, still in flight with an inherited spent marker, behaves as before: a lost turn falls back to Retry, once. - Inside a run, nothing changes: the marker still bounds same-run continuations, together with the provider-attempt restart budget. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Nicky Leach
|
e7e870cfc4 | fix(native): decide warm and adopted Runner process reuse from one key (#20005) | ||
|
DottaandClaude Opus 5.5
|
f6cb3e43f4 |
fix(admission): admit new work independent of other runs' cleanup and accounting (#19944)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Budget admission decides if new agent work can start in a company > - Five admission gates waited on the cleanup or accounting of other runs > - The restart-chaos campaign found that these gates froze companies for up to 10 minutes, caused false budget pauses, denied in-run writes, and dropped wakes, comments and Retries > - The durable ownership plan (Decision 1) removes the company-wide accounting gate. It does not narrow it again > - This pull request removes the five gates. Unfinished accounting now adds its bounded estimate to exposure > - The benefit is that one run's unsettled accounting never stops the other work in its company ## Linked Issues or Issue Description Refs #19831 (plan, section D), #19838 (inventory, finding 3), #19927 (Phase 2 reconciler data), #19497, #19724, #19826. **What happened?** Admission refused new work in these cases, with no reached hard stop and no pause: - a terminal run awaited accounting (`waitingRunCount`); - a native run recovered its accounting (`recoveringRunCount`); - a budget-owned scope hold awaited reconciliation, after the hard stop no longer applied; - a budget input was older than 15 s (`budget_evaluation_work`); - the predecessor run row of a task was missing (`hasPendingRunCleanup` failed closed). **Expected behavior** Admission never waits on the cleanup or accounting of another run. Only a reached hard stop, an explicit pause or an inactive company refuses new work (invariant 3). **Steps to reproduce** Make a company with an active budget policy. Leave one failed run with `cost_accounting_pending = true`. Request work for another agent. On `master` admission refuses with "Company cannot start work while completed runs await accounting." ## What Changed - `budgets.ts`: admission refuses only for a reached hard stop or an explicit pause. The code removes `observedBlocks`, the `recoveringRunCount` check, the stale `budgetHeld` refusal and the 15 s input-age gate. It also removes the unused `inspectInvocationEligibility`. `reconcileScope` no longer returns accounting holds. - `budget-exposure.ts`: removes the `waiting` and `recovering` aggregates and the own-task subquery. The admission query is smaller. Unfinished accounting stays exposure: the larger of the held reservation and the observed cost. - `budget-evaluation-work.ts`: removes `oldestBudgetInputAt`, which nothing reads now. - `execution-blocker.ts`: `hasPendingRunCleanup` returns false for a missing run row. A run row that is deleted removes its finalization coordinator and nulls its lease references, so the row owns nothing. A predecessor that owns a lease or a coordinator still holds the successor of its own task. - `recovery/ownership-invariant-report.ts`: the I3 check no longer mirrors the removed gate. It reports a wake request or a run that was refused with an accounting-hold reason. Any such report is a regression. - `doc/SPEC-implementation.md`: adds the product rule "Admission is independent of cleanup and accounting". `doc/execution-semantics.md` is aligned. ## Verification - New `server/src/__tests__/admission-independent-of-cleanup.test.ts`, on embedded Postgres. It has one case per gate. All six cases fail on `master` and pass here. A seventh case (an explicit pause still refuses) passes on both. One case also asserts invariant 3 together with exposure: two failed runs that each hold a 400-cent reservation fill a 1000-cent budget, and a hard stop still refuses. - Existing tests that asserted the old gates now assert admission (invariants 3 and 4): `closed-native-receipt-accounting`, `budgets-service`, `budget-evaluation-work`, `heartbeat-held-comment-wake`, `ownership-reconciliation-report`. In `heartbeat-held-comment-wake`, each saved comment, Retry, assignment and continuation is delivered exactly once while another run's accounting is pending. Settling accounting later does not deliver it again. - All six files pass locally, one file per process. The server package typecheck passes. - Speed: admission is on the turn path. This change only removes work from it: two correlated aggregates, one own-task subquery and one `budget_evaluation_work` read per policy. No wait or I/O is added before `turn.start`. - Not run here: the chaos waves (K13 storm, K6 crash loop, K4 long holds). The coordinating session schedules them on the shared harness. ## Risks - Behavior change: work starts while accounting for other runs settles. Spend stays bounded by the reservation capacity check and the hard stop, which read the ledger plus observed cost. A run that has no reservation and no observed cost adds zero exposure until its charge commits. This is the gap that Decision 1 accepts. Budget enforcement was already best effort (SPEC 11.6). - A budget-owned pause with no reached hard stop no longer refuses work in budget admission. The paused company or agent is still refused by the existing pause checks until the budget worker lifts the pause. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Nicky Leach
|
94718ce168 | fix(native): continue on a replacement when a retained Runner retires before claim (#19897) | ||
|
DottaandClaude Opus 5.5
|
0189fedffc |
fix(native): bound an idle warm local runner whose controller died (#19879)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A warm runner (`lifecycleMode: warm`) keeps runnerd alive between runs so a follow-up turn starts fast. The follow-up default flip (next PR) makes warm the default for local Codex and OpenCode agents. > - The controller owns a warm runner's idle window. Runnerd ignores the idle timeout itself. > - A graceful controller shutdown suspends idle warm sessions. A controller that dies (SIGKILL, crash) does not. Local runners launch with no process reconnect grace, so the idle runner kept running until its 1-hour connection lease ran out, owned by nobody. > - This pull request gives warm local runners the 7-minute reconnect grace that remote runners already use. > - The benefit: idle runners are reaped by proof within a bounded time, never left orphaned, before warm becomes the default. ## Linked Issues or Issue Description Refs #19874 (warm completion contract), #19846 (TTFT baseline), #19852. **What existing behavior does this improve?** Ownership of an idle warm local runner after its controller dies. **Current behavior** `runnerProcessReconnectGraceMs` is `undefined` for local targets, so runnerd launches without `--reconnect-grace-ms`. An idle warm runner whose controller was killed stays alive until its connection lease (one hour) expires. The restart-chaos harness cannot show this: its K1 kills the controller's whole process tree, runner included. **Proposed behavior** `localRunnerProcessReconnectGraceMs(policy)` returns the remote grace (7 minutes) for a warm policy and `undefined` for per-turn. After the grace, runnerd records `transport_reconnect_grace_exceeded`, stops its provider, keeps durable state and exits. The next run resumes cold. **Reason and benefit** The restart plan requires idle runners to be reaped by proof, never orphaned. Seven minutes is far longer than a controller restart (13–20 s locally) and the long-outage fault holds (3–5 minutes), so a live turn still reattaches. ## What Changed - `native-runner-recovery-contract.ts`: `localRunnerProcessReconnectGraceMs`. - `native-session-executor.ts`: the local runnerd transport passes it as `runnerProcessReconnectGraceMs`. Remote targets keep `null` (lease-owned). - `runnerd-warm-idle-orphan.test.ts` (new): real runnerd and fake Codex. An idle warm runner, after its turn completed, whose controller detaches. - `doc/DEVELOPING.md`: the grace and its bound. ## Verification - `pnpm --filter @paperclipai/paperclip-runner exec vitest run src/live/runnerd-warm-idle-orphan.test.ts`: with a process grace, the idle runner exits with `transport_reconnect_grace_exceeded` and the next transport resumes the same provider session cold. With no grace (`null`, today's local launch), the runner is still alive after the controller left. - `cd server && npx vitest run src/services/native-runtime/native-runner-recovery-contract.test.ts`: warm gets the remote grace; per-turn keeps none; the grace outlasts a controller restart and a long outage. - `tsc --noEmit` passes for `server` and `packages/paperclip-runner`. - Local restart measurement with warm runners (K1/K2, codex, claude, opencode, n = 3 each): every interrupted run recovered, with recovery latency equal or lower than per-turn. The chaos waves with this change are in the default-flip PR. ## Risks - A warm local runner whose controller stays down for more than 7 minutes exits. A turn in progress then resumes through `resume_dead_runner`, the same path as a controller crash that kills the runner. Remote runners already behave this way. - Previous-release story (`rules/upgrade-safe-run-state.mdc`): no persisted state changes. A warm runner launched by the previous release keeps the launch arguments it started with (no grace) until it exits. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edit, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
e2d490c61b |
feat(observability): anchor native time to first token at the accepted wake (#19838)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When someone messages an agent, the reply must start fast. The durable ownership plan (#19831) makes time to first token (TTFT) a hard invariant. > - Phase 0 of that plan records a TTFT baseline per adapter and environment, measured from the moment the message or wake is accepted. > - Today `task.time_to_first_token` starts at the wake comment, the answered question, or the run row's creation. A wake request that was accepted earlier (deferred, coalesced or retried) is not part of the measurement, and the span does not say which start it used. > - This pull request moves the start back to the earliest accepted input and names it on the span. > - The benefit is that the baseline and later budgets measure what the user waits for, with no new work before `turn.start`. ## Linked Issues or Issue Description Refs #19831 **What is the problem?** `task.time_to_first_token` does not include the time between an accepted wake request and the creation of its run row. It also does not record which input it started from. **What is the expected outcome?** The span starts at the earliest accepted input and carries an `anchor` attribute: `comment`, `question_response`, `wake_request` or `run_created`. **Why does it matter?** Phase 0 of the durable ownership plan sets TTFT budgets from this baseline. A start that skips queue time hides the latency that deferred wakes add. ## What Changed - `resolveNativeTimeToFirstTokenStart` (pure) in `native-run-trace.ts` picks the start and the anchor. `anchor` joins the closed span-attribute allowlist. - `readNativeRunWakeRequestedAtMs` reads `agent_wakeup_requests.requested_at` for the run. It never throws. - `native-session-executor.ts` calls it only after the first deliverable text, so it adds no wait before `turn.start`. The span now ends at the text commit time, not after the trace write. - Updated `doc/run-log-events.md` and `doc/observability.md`. ## Verification - `cd server && npx vitest run src/services/native-runtime/native-run-trace.test.ts src/services/native-runtime/native-run-wake-anchor.test.ts` (26 pass). - `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts -t first` (41 pass). The existing TTFT test now asserts the anchor for each ingress. ## Risks - Low risk. This changes only the run-log and OTLP span. It is not Telemetry (AGENTS.md §5.7), and it does not change runtime behavior. - One indexed primary-key read per native dispatch, after the first text. A read failure drops the anchor and keeps the old start. - TTFT values can grow for deferred wakes. That is the intended correction. `startOffsetMs` can be negative when the wake request comes before the trace start. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
84908d176b |
fix(recovery): name the failed check in native_remote_recovery_lease_mismatch (#19839)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Remote native runs execute in a sandbox, and recovery can reattach a run to its original sandbox runner > - Reattachment checks the saved lease, workspace, and runner identity in five places. Each failed check throws the same bare `native_remote_recovery_lease_mismatch` > - On staging, 34 OpenCode runs on 31 tasks failed with this bare error in one evening. The run records could not show which check failed, so the cause could not be confirmed from the run alone > - This pull request makes every producer name the failed step and field with closed values > - The benefit is that an operator and the next investigation can see which reattachment check failed, without new logging or identifiers in the error ## Linked Issues or Issue Description Refs #19399, Refs #19733, Refs #19755, Refs #19779 **What happened?** Remote same-run retries failed with `native_remote_recovery_lease_mismatch` and the task was blocked for reconciliation. The error is thrown by environment acquire, environment resume, workspace preparation, workspace sync, and remote runner verification. All of them used the same bare string. The run's `error` field and its run events did not show which check failed. **Steps to reproduce** 1. Start reattachment for a remote native run whose saved lease, workspace reference, or runner identity no longer matches. 2. Read the run's `error`. 3. It shows only `native_remote_recovery_lease_mismatch`. **Expected behavior** The error names the step and the field that differed, with no identifiers, and existing consumers of the code keep working. ## What Changed - New `server/src/services/native-runtime/remote-recovery-lease-mismatch.ts`: the code constant, a closed `stage` and `field` vocabulary, and `remoteRecoveryLeaseMismatch(stage, field)`. It builds `native_remote_recovery_lease_mismatch: stage=<stage> field=<field>`. - `environment-run-orchestrator.ts` (stages `environment_acquire` and `environment_resume`), `heartbeat/workspace-preparation.ts` (`workspace_preparation`), `native-workspace-sync.ts` (`workspace_sync`), and `verifyRemoteRunnerReattachment` in `native-session-executor.ts` (`runner_verification`) now throw the detailed form. Each check keeps the same condition and order. Only the message changes. - `restart-interrupted-turn.ts` uses the shared constant. It already matches the code exactly or as the `code: ` prefix, so automatic continuation after a reattachment loss is unchanged. No other consumer matches the code. - `doc/execution-semantics.md` documents the detail format. ## Previous-release state This change adds no persisted state and changes no saved identity. Runs that a previous release failed keep the bare `native_remote_recovery_lease_mismatch` in `error`. Runs failed by this release store the detailed form. The only consumer (`isReattachmentLoss` in `restart-interrupted-turn.ts`) accepts both forms, and `workspace-recovery-retry-copyback.test.ts` covers both. ## Verification - Exact-detail assertions in the producer tests: - `server/src/__tests__/environment-run-orchestrator.test.ts`: every scope, status, expiry, provider, lease-ID, cwd, plugin-task, and resume case asserts its stage and field. Against the previous producer, 21 of these cases fail (the message was the bare code). With this change, all 57 tests pass. - `native-session-executor.test.ts`: a replaced sandbox reports `runner_verification`/`provider_lease_id`. A missing durable runner identity reports `identity_run_id` before any sandbox read. - `native-workspace-sync.test.ts`, `heartbeat/workspace-preparation.test.ts`, `native-controller-recovery.integration.test.ts` assert their exact detail. - End to end on embedded PostgreSQL: `heartbeat-native-remote-failure-retention.test.ts` records `native_remote_recovery_lease_mismatch: stage=environment_resume field=lease` on the run when the provider no longer resumes the lease, and it still requests the user's Retry once. `workspace-recovery-retry-copyback.test.ts` adds a detailed-form row that still continues once. - The affected server test files pass locally. `pnpm --filter @paperclipai/server typecheck` passes. ## Risks - Low risk. Only the error text changes. Conditions, ordering, and fail-closed behavior are unchanged. - Logs and run errors show a longer message. The detail has closed values only, so it adds no identifiers or secrets. - A first version of this PR also changed remote retry selection. Review showed that a plain resume in a new sandbox hits the remote workspace's unsaved-work hold, and that a `destroy` disposition does not prove the old sandbox is gone. That change is removed. It needs a separate design. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, run in Claude Code with tool use (shell, file editing, read-only browser access to staging). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
909f4d1a33 |
fix(native): recover exact historical provider pack code (#19779)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native sessions retain the provider pack that admitted their saved
state.
> - An image update can replace that pack's executable, proxy, or
contract.
> - Older sessions can predate the durable three-file launch artifact
receipt.
> - The current image's archive cannot satisfy a different pinned
manifest.
> - This pull request obtains the exact public code from its recorded
release image.
> - Full source, archive, and installed-file checks still control
provider admission.
## Linked Issues or Issue Description
Refs #19558, #19622, #19755.
A saved OpenCode session can fail with
`runner_remote_provider_archive_invalid` after an image update. Its
original provider state and checkpoint remain valid, but its pinned pack
differs from the controller's pack. The old manifest can name a release
that still has the exact code in GHCR. Matching the OpenCode version
does not prove matching proxy or contract bytes.
## What Changed
- Resolve a pinned historical pack from the recorded source-SHA image in
the public Paperclip GHCR repository. Verify the platform manifest, raw
config, source identity, dedicated code layer, selected pack manifest,
archive contents, paths, and hashes.
- Keep a private temporary cache of verified public code. Rebuild it on
an empty host. Bound each source to 4 GiB and three minutes, with two
active imports or cache verifications, 32 distinct queued sources, and
disk-headroom checks. Share warm-cache verification by manifest while
each caller keeps its own cancellation.
- Select this verified archive through the existing uploader. Preserve
installed-pack and preinstalled-pack checks. Refuse unavailable
historical transport before clearing the target. Keep the current-source
identity guard from #19755. Refuse an oversized layer argument before
any target effect.
- Attach the source wait to the actual per-turn startup. Stop, lease
loss, and startup completion cancel it. Keep existing Runner admission
and turn limits.
- Document the registry retention requirement and the distinction
between public code recovery and encrypted provider-state backup.
## Verification
- Genuine baseline RED: the real uploader rejects the selected old pack
before any remote effect. Treatment installs the exact old code through
that uploader.
- Current head: all 19 source/integrity/cache/deadline controls, all 45
archive controls, and 29 actual-caller and manifest/startup controls
pass (93 total). Four review regressions fail against the prior head and
pass with the fixes. The server and colocated test TypeScript graph
passes. Module-boundary and whitespace checks pass.
- Actual public GHCR cold reconstruction: 1,054,587,008 compressed code
bytes, 115.4 s, about 248 MiB maximum owner RSS. All 114 file layers and
the overlay passed source validation.
- Linux Node 24.22.2 fixture: the real historical pack passes the normal
uploader, full installed-file verifier, and pinned OpenCode version
check. It used one transfer, no application startup, no network, and no
provider calls. Revision12b0d2ba passed in 220.2 s with about 282 MiB
owner RSS. The earlier prototype passed in 108.5 s; both measurements
are retained. The three-minute source limit and the fixture's
four-minute limit were unchanged. The later review fixes add shared
cache verification, hash cancellation, and an early size guard. They do
not change extraction or the accepted Linux fixture input.
- Earlier fixture failures are retained: unsafe hardlink classification
was corrected with confined-link controls; a cached image lacked the
required zstd CLI; Docker Desktop bind storage caused GNU tar
directory-identity errors. The passing fixture uses the official Debian
zstd binary and an owned Linux tmpfs target. Production extraction code
is unchanged.
- Fresh Greptile review is 5/5 on
|
||
|
Nicky Leach
|
167e39c4ef | fix(runner): keep warm run.attach events on ordinary accounting (#19707) | ||
|
|
c3180398fa |
fix(native): resume a remote Runner after a settled suspension
fix(native): resume a remote Runner after a settled suspension A scheduled checkpoint retry can retain a remote reattach claim after closing its failed predecessor cleanly suspends and exits the Runner. Resume only that exact settled source after confirmed exit and a fresh identity/state check. Preserve pending work and uncertain-exit fences. Admit the last already-charged provider attempt and share one bounded recovery deadline with the final state read, retaining closed diagnostic labels on read failures. Cover fresh ACPX Runner relaunch on attempts two and three through actual processes and the durable close barrier. Six affected suites pass 1,579 tests, with 14 environment-gated skips; all server typecheck layers pass on Node 25.6.1. The complete provider turn continuation failure remains preserved as follow-up #19751. Refs: #19366 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-Authored-By: Paperclip <noreply@paperclip.ing> Co-Authored-By: Paperclip <Paperclip-Paperclip@users.noreply.github.com> |
||
|
DottaandClaude Opus 5.5
|
24cbea136f |
fix(native): export OpenCode launch artifacts through a confined workspace copy (#19743)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native recovery saves the qualified OpenCode launch files so a cold allocation can resume the same session (#19622) > - Core reads those files from the sandbox with the provider's bulk `syncOut` > - The launch root is under the sandbox account home, not under the workspace remote dir > - Daytona, Kubernetes, and CreateOS confine every export source to the workspace remote dir, so the export always fails > - Every remote OpenCode run on a durable-storage instance now fails 40-90 s after start with `native_session_interrupted` > - This pull request exports a confined scratch copy of the three launch files from inside the workspace runtime dir > - The benefit is that remote OpenCode runs work again, and the provider confinement guard stays unchanged ## Linked Issues or Issue Description Refs #19622 **What happened?** After #19622, every remote OpenCode run on Daytona fails at provider admission. The run error is `Daytona sync source path escapes the workspace remote dir: /home/daytona/.paperclip-runtime/paperclip-runner/provider-artifacts/<digest>`. The error code is `native_session_interrupted`. The last successful run of the affected agent was immediately before #19622 reached the deployment. **Expected behavior** Core saves the qualified launch files and the run continues. The provider confinement guard still refuses any export source outside the workspace remote dir. **Steps to reproduce** 1. Configure an OpenCode agent on a Daytona environment with durable harness storage. 2. Start a run. 3. The run fails when Core captures the provider artifacts, before the first turn. ## What Changed - `captureRemoteNativeProviderArtifacts` now takes the workspace `remoteCwd`. - Before the export, a sandbox command copies the three launch files into a new scratch directory at `<remoteCwd>/.paperclip-runtime/paperclip-runner/provider-artifact-export/<uuid>/`. The command refuses a symlink or a non-regular file. It uses `cp -P`, so it never follows a link. - `syncOut` reads from that scratch copy. A `finally` step removes the scratch copy after success or failure. A failed removal is logged and does not replace the transfer result. - The launch root does not move. OpenCode still binds to the same absolute paths. Core still verifies each downloaded byte against the manifest SHA-256 values. - `.paperclip-runtime` is already excluded from workspace sync, so the scratch copy never enters the user workspace. - The Daytona `assertConfinedSandboxPath` guard is not changed. ## Verification - New regression `server/src/services/native-runtime/native-provider-artifact-capture-confinement.test.ts`. It runs the real sandbox commands on a local directory tree that has a home-rooted launch root and a separate workspace. Its `syncOut` applies the same workspace confinement rule as the Daytona provider. - Before the fix: it fails with `Daytona sync source path escapes the workspace remote dir: .../home/daytona/.paperclip-runtime/paperclip-runner/provider-artifacts/dfe9e4b1d660/bin/nodeCommand`. This is the staging error. - After the fix: it passes. The published bytes match, every export source is under the workspace runtime dir, and the scratch directory is empty after the capture. - Safety cases: a failed transfer removes the scratch copy and does not publish. A launch file replaced by a symlink is refused before any export, and nothing is published. - `native-session-executor.test.ts`: the durable OpenCode admission case now asserts that the capture receives `remoteCwd` and that every `syncOut` source is under `/workspace/`. - `pnpm vitest run` for the capture, confinement, remote OpenCode artifact, and session root entries tests: 14 passed. - `vitest run native-session-executor.test.ts -t "opencode|provider artifact|durable"`: 62 passed. - `pnpm --filter @paperclipai/server typecheck`: passed. - Not run: a live Daytona run with this revision. The fix is not deployed to staging. ## Risks - Low risk. The capture now runs two short `sh` commands in the sandbox and holds a second copy of the launch files during the transfer. The files are bounded by the existing 512 MiB limit and are deleted afterwards. - The scratch copy uses `cp -P`, `mkdir`, and `rm -rf` from POSIX `sh`. The Daytona image already uses these tools for the workspace sync. - If the sandbox stops between the copy and the removal, a scratch directory can remain under `.paperclip-runtime`. Workspace sync excludes that directory. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edit, and read-only browser access to staging). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
fc7e3a6503 |
fix(recovery): never replace a native session after a deterministic refusal (#19733)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native agent runs keep a saved provider session, and same-run
recovery resumes it after a failure
> - When resume fails and the checkpoint allows it, recovery archives
the saved continuity state and opens a fresh replacement session
> - Recovery used that path for every resume failure, including
deterministic refusals that left the saved state unchanged and repeat on
every retry
> - In five staging tasks a refusal (a protected launch argument, or a
predecessor tool call with no result) escalated into a replacement. The
remote archive move failed part way, the next attempt found no runner
state, and each task was blocked for workspace repair
> - This pull request stops before replacement after a closed-class
refusal, keeps the saved state, and hands the run to an operator with a
named reason
> - The benefit is that a deterministic refusal can no longer damage
saved state or open a replacement over an unsettled external action
## Linked Issues or Issue Description
Refs #19659, Refs #19516, Refs #19334
**What happened?**
Five Codex tasks on Daytona sandboxes failed on staging. The first
attempt failed with a deterministic refusal:
- `run.attach cannot change protected launch arguments` (four tasks,
fixed separately in #19659), or
- `native_runner_source_semantic_reconciliation_required` with a
predecessor `paperclip_call_assigned_tool` call still pending (one
task).
The same-run retry then opened a replacement session.
`archiveContinuityState` failed with
`runner_continuity_break_archive_failed` and no detail. The third
attempt failed with `runner_remote_state_unavailable: exit=1`, and the
task was blocked with "Workspace repair required".
**Steps to reproduce**
1. Save a native session whose checkpoint policy is
`allow_replacement_after_resume_failure`.
2. Make the resume fail with a refusal that does not change state (an
attach refusal, or a pending tool call).
3. Same-run recovery opens a replacement session and archives the saved
continuity state.
**Expected behavior**
A deterministic refusal stops the run for an operator. The saved state
stays intact. A run with a pending side-effecting call never reaches
replacement. Transport drops keep the existing retry behavior.
**Root cause**
`executeNativeSession` checked only the checkpoint's replacement policy.
The closed failure class (#19334) only had an effect when replacement
was forbidden. Attach refusals and pending tool calls had no class at
all, and the source-reconciliation refusal was a plain `Error`.
## What Changed
- Runner (`provider_backend.rs`, `durable/mod.rs`): `run.attach`
refusals that would change the durable Codex profile or a protected
launch argument carry the closed classification
`provider_attach_refused`. An attach blocked by
`durable_pending_tool_calls` carries `provider_side_effect_pending`.
These refusals already leave the saved state unchanged.
- Contracts (`harness-driver.ts`): new closed classes
`PROVIDER_ATTACH_REFUSED` and `PROVIDER_SIDE_EFFECT_PENDING`. A typed
`NativeRunnerSourceReconciliationRequiredError` replaces the plain
`Error` at the five `native_runner_source_*_reconciliation_required`
throw sites. `runnerRecoveryFailureClass` reads the authenticated
command-result classification or the typed error, never message text.
- Runtime (`native-session-runtime.ts`): after either class, recovery
never opens a replacement session, whatever the checkpoint policy.
- An attach refusal ends with the existing
`NativeProviderStateIncompatibleError`, which maps to the existing
operator path `native_provider_state_incompatible`.
- A pending side effect ends with the new
`NativeProviderSideEffectPendingError`.
- Server (`native-session-executor.ts`, `native-session-resume.ts`,
shared `EXECUTION_RECONCILIATION_CAUSES`):
`native_provider_side_effect_pending` is a permanent failure and an
execution-reconciliation cause, so the board gets a reconciliation
(record whether the action ran) instead of a generic Retry. The
reconciled continuation does not rebind the saved session, which still
refuses attachment. It starts a fresh provider session in the same
workspace. The saved session and workspace are unchanged.
- Archive (`archiveContinuityState`): a failed remote move now records
the bounded (512 characters), redacted exit status and output in
`runner_continuity_break_archive_failed: exit=… timedOut=… <output>`.
The remote move already ran before the local move on `master`, so a
failed remote move leaves the local state in place.
`restart-interrupted-turn.ts` accepts the detailed message for its
reattachment-loss retry.
- Docs: `doc/execution-semantics.md` and
`packages/paperclip-runner/docs/durable-recovery.md` describe the new
rule.
## Verification
Red before the fix, green after (each test first run against unchanged
source from `origin/master`
|
||
|
Nicky Leach
|
b21884bdcf | Release and resume retained plugin Runner sessions in Core (#19611) | ||
|
DottaandClaude Opus 5.5
|
4612fa9fb4 |
fix(native): restart a drained Daytona run whose runner opened its session (#19711)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs on Daytona start a `paperclip-runnerd` process in a sandbox. A graceful shutdown (every rolling deploy) suspends live native runs "without cancelling provider work". The next boot then decides how each run continues. > - #19583, #19616 and #19684 let restart recovery start a remote bootstrap again when durable evidence proves that no provider work exists. > - A chaos replay found another window. The drain suspended a remote run just after its sandbox runner opened the provider session, before the controller sent a turn and before a session snapshot existed. > - Recovery accepted only an unfinished `session.open` in the controller checkpoint, so it blocked the task as ambiguous. A Retry recovered it, but automatic continuation was lost. > - A related replay showed that the same-run bootstrap reuses the sandbox. The interrupted runner can still hold the runner ingress port there (`runner_ingress_bind_conflict`). > - This pull request proves the "session opened, no turn" case and stops the interrupted runner before relaunch. > - The benefit is that a deploy during Daytona startup continues automatically, without a port conflict. ## Linked Issues or Issue Description **What happened?** A chaos replay (ACPX Claude on Daytona) sent SIGTERM while a remote run was starting. The drain recorded `restart_suspended`. The dying controller still launched the runner and finished `session.open`. On the next boot, restart recovery returned `ambiguous_remote_recovery_evidence` with disposition `blocked`. The decrypted controller checkpoint of that run showed: - `run.prepare` completed and `session.open` completed (provider session opened). - Committed events `session.started`, `session.capabilities.updated` and `session.goal.snapshot` (`ackedSourceSeq: 3`). - No turn command. No run-profile session snapshot and no projected provider events. On the next run in that replay, a same-run bootstrap relaunched the runner in the same sandbox (one lease for all attempts). It failed with `runner_ingress_bind_conflict: failed to bind listener: Address already in use`, because the interrupted runner from 51 s earlier was still listening there. The bounded same-run retry recovered it on its last attempt. **Expected behavior** When the durable controller record proves that no turn was ever sent, recovery starts the bootstrap again. It first stops the interrupted runner in the reused sandbox. **Steps to reproduce** 1. Start a native run on a Daytona environment. 2. Send SIGTERM to the control plane while the runner opens its provider session. 3. Restart the control plane after the controller lease expires. 4. See that restart recovery blocks the task with `ambiguous_remote_recovery_evidence`. Root cause: in `server/src/services/native-runtime/native-restart-recovery.ts`, the remote checkpoint proof (`remoteBootstrapProvenIncomplete`, #19583) required an unfinished `session.open` with no committed events, and no provider events at all. A completed `session.open` with only session-open events had no rule, so it fell through to `ambiguous_remote_recovery_evidence`. Refs #19583, #19616, #19684, #19524 ## What Changed - `native-bootstrap-proof.ts`: new `runnerdStateHasNoProviderTurn` proof. The state must hold only `run.prepare` and `session.open` (pending or completed) and no warm, staged or adopted state. Every committed event must be a session-open event, with contiguous sequence numbers from 1, so no event is hidden. Only the controller issues turn commands, and it writes each one to Postgres before sending it. - `native-runner-checkpoint.ts`: `nativeRunnerCheckpointRemoteBootstrapProof` replaces `nativeRunnerCheckpointProvesIncompleteBootstrap`. It also reads a bounded number of `event:N` parts and returns which proof holds. It still rebuilds the state in memory. - `native-restart-recovery.ts`: a remote run can now claim `bootstrap_incomplete` with reason `remote_session_opened_without_turn`. Projected provider events may only be session-open events (`harness.ready`, `session.started`, `session.resumed`, `session.updated`), and only together with the stronger session-opened proof. A turn or provider event still blocks. - `native-session-executor.ts`: - For a remote sandbox target, the executor accepts the same proof on the restored record. - Before the same-run bootstrap relaunches, `stopStaleRemoteRunnerBeforeBootstrap` stops the interrupted runner in the reused sandbox. It uses the verified identity marker and process check from reattachment, which signals only a process whose identity matches this run: SIGTERM, a wait of up to 10 s, then SIGKILL. - The stop is best effort. If no prior runner can be verified, nothing is stopped, and the bounded same-run retry still handles any conflict. - `doc/run-log-events.md`: document the reason and the stop. ## Verification - Red, then green: `cd server && npx vitest run src/services/native-runtime/native-runner-restart-recovery.integration.test.ts -t "drained remote"`. Before the fix, the opened-session case (with a projected `session.started`) returned `blocked` / `ambiguous_remote_recovery_evidence`. - These keep the hold: a `turn.start` command, a committed `turn.started`, an acknowledged event that is missing from the record, and a projected `turn.started`. The executor test restores the checkpoint and checks that the local proof is false and the remote proof is true. - Regression: `native-runner-restart-recovery.integration`, `native-session-executor`, `native-restart-recovery`, `native-bootstrap-retry.integration` and `heartbeat/recovery`: 1460 passed, 14 skipped. - Stale runner stop: controlled sandbox-runner tests (`native-session-executor.test.ts`, "stale runner stop before a same-run remote bootstrap") run the real reattachment identity and process scripts against a fake sandbox runner: - a verified runner that exits on SIGTERM gets SIGTERM only; - a runner that ignores SIGTERM gets SIGTERM, then SIGKILL; - a runner with another identity gets no signal; - no live prior runner means no signal. - Live Daytona proof (wave `w-20261107T021837-s29`, ACPX Claude, `--faults K2 --triggers time,result`, 4 tasks, concurrency 1). The image was built from current master (`paperclip-daytona-runner@sha256:71f711ab…`), and its content id matches this branch. - All 4 tasks crossed a graceful restart, and all passed R1. Two of the K2s fired on the `result` trigger, during a running remote turn. - The K2 during a turn reattached to the live remote runner (`reattach_remote_runner`, generation 2), as designed. - Task t01 failed R6 with an orphan Daytona sandbox that no lease references. The graceful restart came during sandbox creation. This is a separate signature, outside this change. - The timed faults did not land in the exact "session opened, no turn" window. The integration tests cover that window with the evidence from the saved replay database. ## Risks - The proof relies on two facts: only the controller issues turn commands, and the controller writes each command before sending it (`#sendJson` waits for the checkpoint flush). A turn always leaves a `turn.start` command in the record. - The provider session opened in the old sandbox runner is not resumed. The new bootstrap opens a fresh session. No turn ran in the old one, so no work is lost. - The runner stop sends signals inside the run's own sandbox. It signals only after the existing reattachment identity and process checks pass. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, GitHub CLI) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
18be7ac696 |
fix(server): retire idle native sessions before tenant sleep (#18988)
## Thinking Path > - Paperclip lets people manage AI agents and their work. > - Native agents can retain a warm session between useful turns. > - That owner must close and save its files before the tenant can sleep. > - A failed close must keep the owner visible until retirement succeeds. > - This PR prepares idle resources under the exact current admission hold, then takes a fresh safety snapshot. > - Busy turns, uncertain closure and changed holds still prevent sleep. > - CI also exposed a chat authorization race. The fix repeats full authorization after the exact delivery commits. ## Linked Issues or Issue Description Refs #18639 for local work and incoming wake boundaries. Related #18829 covers suspension receipts after admission. **What happened?** Idle preflight did not retire warm Native owners. Three failed-close paths could forget an owner without proving that its provider stopped. CI also found a false chat rejection when a delivery changed from processing to processed between two validation reads. **Expected behavior** Retire only idle owners under the exact bounded hold. Wait for closure and file collection. Retain failed owners. Scan safety after preparation. Keep diagnostic reads passive. A newly committed chat delivery must pass the full current authorization checks before attestation succeeds. **Steps to reproduce** Finish a warm Native turn. Start an idle drain and request safety with its owner ID. Delay or reject provider closure, or replace the hold during preparation or scanning. The report must keep the instance awake. For the chat race, commit the exact delivery between the first binding read and its fallback read. **Paperclip version or commit** Current head `7f6f02b611dcf5a9a2b05173a44e9631eaf78058` normally integrates actual master `1aa0c3639714f4976cfa56f95aae6e2b7d1235ac`. The merge preserves every unrelated master path. Fresh exact-head CI and complete review remain required. ## What Changed - Retain uncertain Native owners after failed restart, idle timeout and failed-turn closure. - Prepare Native retirement through the existing heartbeat facade. Reuse provider closure and stopped-file collection. - Track preparation under the exact idle owner and expiry. Reject new activity, changed holds and expired holds before and after awaited work. - Take the final durable and local safety snapshot after preparation. Keep diagnostic inspection passive. - Cover ephemeral workspace retirement and useful next use, supported plugin drain and SDK persistence, busy owners, failed closure and hold replacement. - Keep dedicated recovery storage explicit. Use the same validated options for support and storage. Probe-free helpers retain the original implementation bodies and preserve default callers. - Complete the Native fixture through real retirement and budget evaluation. Keep its rejection, file, credential, accounting and strict NONE assertions. - Repeat the complete chat authorization once if the fallback sees the exact batch now processed. Preserve owner locks, principal and binding checks, denial handling and the original retry deadline. ## Verification - The support-mode regression reproduces one failure with four passes under the old policy, then passes all five controls. It retains prior allocation and cleanup cases. The explicit mode does not probe embedded PostgreSQL or use an admin alias. - The deterministic chat race baseline reproduces two authorized failures with all four current-authority denial controls passing. Both Slack and Discord use real PostgreSQL and the real authorization function. The negative setup commits before the attestation transaction; it does not claim to revoke access through a row lock. - The product fix passes normal own App declaration emission with no public declaration changes, plus both current normal owning compiler programs (399 and 611 roots). The same six controls pass after the fix. The full original file plus these controls passes all 30 tests with no skips. - The original controlled Native continuation passes: one test, no failures or skips, 1.718 seconds in its body and 23.138 seconds in its launcher. The original 40-second case and 180-second outer bounds are unchanged. It verifies held closure, strict NONE, an ordinary authorized comment, saved session/files, useful output, encrypted credentials, accepted results and settled accounting. All ten observed processes and groups close; worker connections and listeners reach zero. A separate post-owner audit finds zero sessions; the successful owned database is normally removed. - Earlier preparation, HTTP, lifecycle, workspace and plugin controls remain source-specific evidence. They are not a claim that every final-tree test ran locally. - The original CI assertion log is retained. The controlled race proves a possible cause; the original CI SQL schedule was not captured. Two other jobs lost their self-hosted runners and provided no test verdict. Earlier asset timeouts, readiness failures and unexplained later recovery remain separately recorded. No timeout, failure predicate or authority check was weakened. Focused public commands: ```sh pnpm exec vitest run server/src/__tests__/idle-sleep-preparation.test.ts server/src/__tests__/instance-settings-routes.test.ts --maxWorkers=1 --no-file-parallelism pnpm exec vitest run server/src/__tests__/heartbeat-reviewed-chat-binding.integration.test.ts --maxWorkers=1 --no-file-parallelism ``` Local database qualification uses fresh owned test databases on existing PostgreSQL, bounded pools and one worker. The local allocation adapter changes only database allocation for this test module; it runs normal migrations and client cleanup. It is not part of the published source. No new cluster is created. Local acceptance measured the `0a69efa6` integration with the same committed authorization bytes. The later normal master merge adds startup-timeout classification and recovery changes. Those paths are outside the controlled healthy-backend witness; retirement and authorization hunks are preserved. Exact-head remote CI at `7f6f02b611dcf5a9a2b05173a44e9631eaf78058` has completed with 56 successful contexts and four skips. The complete current Greptile summary is5/5 with no new actionable findings; all review threads are resolved. Ponytail’s current turn is explicitly paused without a current-head verdict; earlier actual findings are fixed and accounted for. No whole final-tree local repository build/test run, live-provider restore or deployed suspension is claimed. Actual master subsequently advanced to `d5c4f7dd02d408a261cedd9a1926b9900a4aedc4`. Its local normal-merge projection with this reviewed head preserves all20 owned paths and every unrelated master blob and mode. Current Runner TypeScript, App declarations and their full owning compiler programs pass. Focused Native cleanup and warm-continuation composition checks are being completed before merge; the new upstream suspension-retry behavior is not inferred from the older controlled-backend witness. The remote PR head remains unchanged. ## Risks - Privileged idle preparation can close idle warm Native sessions. It requires the exact current hold. - Failed closure retains its owner and can keep the tenant awake until normal retry succeeds. - A changed hold cannot authorize sleep. Started cleanup remains tracked until it settles. - The chat repair can perform one extra full authorization read. It keeps the existing deadline and does not grant authority from the fallback query. - Unsupported resources and active work retain their existing blockers. This PR does not certify every transport, plugin or deployed host. ## Model Used OpenAI `gpt-6.1-sol` produced the delegated changes and verification. OpenAI Codex, based on GPT-6, performed managing review and integration. Both used repository tools and code execution. The coordinator's exact backend snapshot and context-window limits were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green at exact head `7f6f02b611dcf5a9a2b05173a44e9631eaf78058` - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups at the exact current head - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Codex <noreply@openai.com> |
||
|
DottaandClaude Opus 5.5
|
30857db55e |
fix(native): recover a run killed between runner launch and its identity record (#19694)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs start a local `paperclip-runnerd` process. The server records the request to start it, then records its process identity after it spawns. > - If the server is killed between those two records, there is no process ID to check after the restart > - Restart recovery then restarted the bootstrap, but the executor failed with `durable_identity_unreadable` because the runner never wrote its state > - The user's Retry then waited forever with "Retry is waiting for Paperclip to verify that the previous execution stopped", because the stop check needs a process ID > - This pull request restarts that bootstrap and proves on the host that no runner for the run exists > - The benefit is that a restart in the runner launch window no longer loses the task or leaves Retry permanently stuck ## Linked Issues or Issue Description **What happened?** In restart-chaos testing, an ACPX Claude run was created and the server was SIGKILLed 1.3 s later, right after `native.process_start_requested` and before the runner identity was recorded. On restart, recovery claimed `bootstrap_incomplete` (`dead_runner_before_provider_identity`). The executor then threw `runner_state_identity_mismatch: durable_identity_unreadable`: the state root existed, but `control-plane/control-plane-state.json` had never been written. The run failed with `native_continuation_requires_reconciliation`. A Retry was accepted (HTTP 200), but no run was ever created. The issue stayed blocked with `retryWait.reason=process_identity_missing` for over 20 minutes and through another restart. **Expected behavior** When the runner never wrote state, no provider started, so the bootstrap can start again. A Retry should be admitted once the host proves that no runner for the run exists. **Paperclip version or commit** `master` at `fb3cf1a072`. **Steps to reproduce** Create a native local task and SIGKILL the server about 1 s after the run starts, after the runner launch is requested and before its identity is recorded. Restart the server, then click Retry. ## What Changed - `migrateRunnerdStateRootForExecution` (`native-session-executor.ts`): for a `bootstrap_incomplete` restart on a local target, a root is quarantined (kept for inspection) and the bootstrap starts again only when both of these hold: - the root has no `control-plane-state.json`; - no live runner for the run exists on this host. A state file that exists but cannot be read, or a runner that started but has not connected yet, still fails closed with `durable_identity_unreadable`. - `unidentifiedLocalRunnerProvenAbsent` (`native-session-executor.ts`) proves that a runner is absent when both of these hold: - no host process carries the run's `--run-id <runId>` argument (`paperclip-runnerd` is always started with it, detached and unreferenced); - no saved control-plane state for this run exists, in the canonical root or in quarantine. A provider starts only after its runner connects and writes that state. - `native.process_start_requested` now records the launching `hostname`. - `recordUnidentifiedLocalRunnerStop` (`native-local-process-stop.ts`) applies the proof only when all of these hold: - the run is native; - the run has no process ID; - its latest launch event is `native.process_start_requested`, from this host; - all of its leases are local. A launch recorded without a host stays blocked. - The process listing runs asynchronously with a 5 s deadline. A failed or slow listing proves nothing. It then writes `native.local_process_stopped` with null process IDs and `basis: "unidentified_runner_absent"`. A later launch request invalidates it as usual. - The continuation and Retry gate (`explicit-native-continuation.ts`) uses this proof before returning `process_identity_missing`. A dry run checks the proof but records nothing. - `doc/run-log-events.md` documents the new receipt basis. ## Verification - `native-session-executor.test.ts`: - The bootstrap restart with absent state was red before the fix (`durable_identity_unreadable`) and is green after it (root quarantined, backend admitted). - Unreadable state, and a live runner that has not connected, still reject. - Proof cases: absent → true; a live process with `--run-id <runId>` → false; this run's saved state → false; another run ID → false. - `explicit-native-continuation.test.ts` (embedded Postgres, real admission): - The `absent` and `dry_run` cases were red before the fix (blocked) and are green after it (admitted). Only the real admission records the receipt. - The `live_runner`, `saved_state`, `foreign_host` and `unbound_launch` cases stay blocked. - Broader local run after the review fixes: `explicit-native-continuation`, `native-session-executor`, `native-recovered-owner-followup`, `native-runner-restart-recovery.integration` and `task-retry` gave 1836 passed, 0 failed. - A chaos replay of the original task (`--replay w-20261106T232854-s7103/t26`) passed (R1 PASS). Its kill landed after the runner identity was recorded (`resume_dead_runner`), not in the launch window, so the tests above are what reproduce that window. - `tsc --noEmit` for `server` passes. ## Risks - The proof depends on `ps -axww` listing every process on the launching host. If `ps` fails or times out, the proof fails closed (still blocked). Remote runners and other hosts are excluded. - A runner that was spawned but has not yet written its state could be mid-start when admission runs. It would still carry `--run-id` in its arguments, so the proof refuses. - Only the bootstrap path with no state file changes. Ownership, checkpoint and reattach checks are unchanged. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test execution). Extended reasoning enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
803cb3bfe3 |
fix(native): re-bootstrap an OpenCode runner drained before its first turn (#19704)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs start a local `paperclip-runnerd`. After a restart, recovery may start the bootstrap again when the durable state root proves that no provider work started. > - #19665 added a proof for a graceful drain that lands after `session.open` started a fresh session and before the first turn. It requires every recorded provider process to have exited. > - OpenCode records provider startup ownership twice during `session.open`. The first record comes before launch, with `processId: null` and `processGroupId: null`. > - The proof treated `processId: null` as a process it could not prove gone. An OpenCode run drained in this window still failed with `runner_state_identity_mismatch: bootstrap_not_proven_incomplete`. > - This pull request treats a null process id as "no process yet". The benefit is that a graceful restart during OpenCode bootstrap continues automatically, as it already does for ACPX. ## Linked Issues or Issue Description Refs #19665, #19524 **What happened?** A restart chaos wave sent SIGTERM to the control plane about 11 s after an OpenCode run started. The drain suspended the run. At the next boot, recovery verified that the local runner had stopped and chose `bootstrap_incomplete`. The executor then failed the run with `adapter_failed` and `runner_state_identity_mismatch: bootstrap_not_proven_incomplete`. The kept state root had: - `run.prepare` and `session.open` completed, with a freshly started session (`status: "started"`); - only `session.started`, `session.capabilities.updated` and `session.goal.snapshot` committed; - two `harness.diagnostic` events of type `provider_startup_ownership` in the runner outbox, the first with null process ids and the second with the server pid and process group; - no turn command, and the OpenCode server process had exited. **Steps to reproduce** 1. Start an OpenCode task on a local runner. 2. Send SIGTERM to the control plane after `session.open` completes and before the first turn. 3. Let the control plane restart. **Expected behavior** Recovery starts the bootstrap again, and the task continues. **Actual behavior** The run fails. The task needs a manual Retry. ## What Changed - `native-bootstrap-proof.ts`: the process-id collector skips `processId`, `pid` and `processGroupId` values that are `null`. A null value records a launch that has not produced a process yet. Any other non-number value still fails the proof, and every numeric id must still be proven gone. - Tests: a red test with the OpenCode startup-ownership shape, plus counter-tests for a live OpenCode server process, a non-numeric process id, and a started turn. ## Verification - Red before the fix: "allows a fresh bootstrap after a drain once OpenCode opened its session before the first turn" failed with `expected false to be true`. Green after the fix: the `native incomplete-bootstrap evidence` block passes 36/36. - I ran the proof read-only against kept real roots: - the OpenCode drained root now returns `true`; - an OpenCode root from the same wave where a turn ran (108 events) still returns `false`; - the three roots from #19524 and #19665 still return `true`. - `native-runner-restart-recovery.integration.test.ts` and `native-restart-recovery.test.ts`: 134 passed. `tsc --noEmit -p server/tsconfig.json` passes. - Live replay: see the PR comment. ## Risks - Low. The change only skips null ids. Numeric ids still need a proven exit. Any turn, provider output, resumed session, or remote runner still fails closed. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude Code with tool use (shell, file edits, test execution). Default context window. Extended reasoning was available. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
9a52a2dfeb |
fix(native): retry ACPX session.open startup timeouts and give cold Claude a 60 s bound (#19661)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runs start an ACPX provider through Runner. `session.open` opens the provider session before any prompt. > - On a loaded host, a cold ACPX Claude `session.open` took longer than its 30-second bound. The controller reported `runner_local_connect_failed: PRP command session.open timed out`. > - The server classified this as the generic `native_session_interrupted` with `ambiguous_state`, so the task was blocked at once. A person had to click Retry, sometimes several times. In one restart-chaos storm, one task needed three Retry clicks, and on another task both Retry runs timed out too. > - The server already has a bounded same-run bootstrap retry for failures that come before provider work (#18854). It did not recognize this timeout. The 30-second bound also cut off normal slow startups. > - This pull request gives cold ACPX Claude a 60-second `session.open` bound, like Pi and OpenCode (#19487). It also retries a startup `session.open` timeout automatically on the same run, from clean state. > - The benefit is that a slow cold start on a busy host completes, or retries itself, instead of waiting for a person. ## Linked Issues or Issue Description Refs #18854 (bounded bootstrap retry for definite `session.open` failures), #19487 (60-second cold OpenCode admission), #19665 (the bootstrap proof that this change reuses). **What happened?** ACPX Claude runs failed with `runner_local_connect_failed: PRP command session.open timed out` about 35 to 40 seconds after they started. The coordinator recorded `native_session_interrupted` with `recoveryMode: "ambiguous_state"` and `providerSessionEstablished: false`. The task was blocked with a board-owned action. No provider event and no checkpoint existed. The host 1-minute load was 17 to 44, sometimes with no restart nearby. **Expected behavior** A normal slow cold start completes. A `session.open` that still times out during runner startup, with no provider event, no checkpoint and no turn, retries on the same run with backoff. At most two automatic attempts follow the first. A run where provider work may have started is never retried automatically. **Steps to reproduce** 1. Run ACPX Claude tasks on a host with a 1-minute load of about 25 or more. 2. Observe a run fail after about 35 to 40 seconds with `PRP command session.open timed out`. 3. Observe the task blocked with `ambiguous_state` and a manual Retry. **Paperclip version or commit** `master` at `6c7cc2c5d6`. ## What Changed The bound (evidence below): - `coldAdmissionTimeoutMs` gives cold ACPX Claude 60 seconds (live adoption keeps 30 seconds). The runner's Rust sidecar `session_open_timeout` and the sidecar's provider handshake (`acpxSessionHandshakeTimeoutMs`) give Claude and Pi 60 seconds too. All three bounds must agree, or the shortest one fails first. Claude's warm run attachment uses the same cold budget, like Pi. The automatic retry: - `nativeSessionFailureSourceCode` returns `provider_session_open_timeout` only for exactly `<startup failure code>: PRP command session.open timed out`. The startup failure codes are `runner_local_connect_failed`, `runner_direct_wss_failed` and `runner_ingress_unavailable`. The transport uses them only before startup completes. After startup the code is `provider_transport_failed`, which is unchanged. - `nativeProviderRecoveryEvidence` treats the cause as pre-session. Durable evidence still decides: a bootstrap retry needs no provider event (`session.started`, `turn.started` and the others) and no checkpoint. The existing budget applies: three attempts with backoff of about 30 seconds, then about 60 seconds. When the budget is used, the task is blocked with the existing "Provider startup failed on 3 attempts before any provider work began" action. - A timeout is indeterminate: the runner can open a provider session after the controller stops waiting. Live runs showed both leftovers, so the retry now starts from clean state: - `prepareNativeSessionOpenTimeoutRetry` (server) runs when the failure settles. It quarantines a local runner root only when the restart-recovery proof (`runnerdStateProvesIncompleteBootstrap`, #19665) shows that no turn began and every recorded process is gone. Any other root, and any remote runner, stays `ambiguous_state` as before. - `resetAbandonedBootstrapRecord` (runner ACPX adapter) runs on a fresh open (no runner-recorded identity). If ACPX has a saved record for the session key with no messages, no prompt and no events, it is marked `reset_on_next_ensure`. Claude saves nothing before the first prompt, so reusing such a record fails the first turn with `session_resume_required`. A record that holds any conversation is unchanged. - A chat retry of an exhausted `provider_session_open_timeout` run is accepted like the other pre-provider causes. - `doc/execution-semantics.md` and the Runner README describe the cause and the bound. ## Verification **Measured startup distribution** (ACPX Claude `runner.session.bootstrap` / `runner.session.startup` spans from restart-chaos run events, 1-minute load 8 to 44): | Source | Bound | Successful cold bootstraps (s) | Failures | | --- | --- | --- | --- | | Earlier storm evidence | 30 s | 2.9, 4.7, 7.4, 8.6, 8.7, 26.2, 27.4, 27.6, 29.2 | startups ended at 34.9 to 40.3 s (the 30 s `session.open` bound plus spawn and `run.prepare`) | | This branch, wave s3601 | 60 s | 6.3, 7.8, 8.1, 13.4, 14.4, 22.3, 22.8, 22.9, **37.3** | 0 `session.open` timeouts | | This branch, wave s3602 | 60 s | 13 samples, p50 10.5, p90 17.9, max 25.7 | 0 `session.open` timeouts | | This branch, wave s3610 (final head, load 28 to 42) | 60 s | 10.4, 11.3, 11.7, 12.0, 13.3, 14.2, 29.6, **30.4**, **30.9** | 0 `session.open` timeouts | Under load, successes clustered just under 30 s and failures sat just above it, so the 30 s bound was cutting the distribution off. With 60 s, cold starts of 30.4, 30.9 and 37.3 s succeeded that the old bound would have failed. 60 s is about 1.6 times the largest observed success, and it matches Pi and OpenCode. **Live restart-chaos waves** (local, ACPX Claude only, 4 in flight, `--rc06-masked`, real provider, K1/K2 restarts): - s3601 (8 tasks, load 19 to 32): 7/8 PASS. No `session.open` timeout. The one failure is an R3 Retry signature after K1 (`native_restart_recovery_blocked`), not this cause. - s3602 (8 tasks, load 17 to 39): 8/8 PASS. No `session.open` timeout. Three Retry clicks were for `native_provider_terminal_failed` (`provider_turn_lost_on_restore`), a different restart signature. - Forced timeout, s3609 (4 tasks, B5, K2): a local, uncommitted fault injection gave every other cold startup a 1.5 s `session.open` deadline. Result 4/4 PASS. Run 4fe6537a timed out, retried itself on the same run, and then succeeded through four K2 restarts with no person involved. Run 02799a85 hit the injection three times in a row; the budget stopped it after attempt 3, and one Retry completed it. - Before the two clean-state fixes, the same forced wave (s3607 and s3608) showed the retry reusing leftover state: `turn.start` failed with `session_resume_required`, and then "ACPX session is not open". Those runs are the red evidence for the two clean-state fixes. - Final head, s3610 (8 tasks, K1/K2, no injection, load 28 to 42): **8/8 PASS**, 0 `session.open` timeouts. The one Retry click (t06) was for `provider_turn_lost_on_restore`, a different restart signature. **Tests:** - Red before the fix: `native-startup-retry.integration.test.ts -t timeout` (embedded PostgreSQL, real executor, real resume claim). The 3 retry cases failed with `native_session_interrupted` / `ambiguous_state`. - Green: `native-startup-retry.integration.test.ts` passes 36/36. New cases: - for each startup failure code, a timeout retries on the same run with backoff, keeps the task `in_progress` with an agent-owned `resume_native_run` wake, and blocks after attempt 3 with a board-owned action; - no automatic retry when `session.started` or `turn.started` is durable, when the code is `provider_transport_failed`, or when another command timed out; - a real runner root with an opened session and no turn is quarantined; a root with a turn command stays `ambiguous_state` and intact; a remote runner is never prepared. - `codex-runtime-adapter.test.ts`: 113 pass. New cases: a 45 s delayed handshake is admitted for Claude and rejected for Codex; a never-prompted record is reset only on a fresh open; records with messages, a prompt or events are kept. - `abandoned-bootstrap-record.test.ts` (real `createAcpRuntime` with `createRuntimeStore` and a controlled ACP peer that, like Claude, cannot load a never-prompted session): without the reset, a fresh open reuses the record and the first turn fails with SESSION_RESUME_REQUIRED after `session/load`. With the reset, the provider receives `session/new` and the turn completes. - `vitest run src/drivers/acpx src/cli` in the runner: 107 files, 1,935 tests pass. - `native-session-executor.test.ts -t "session.open|startup|bootstrap"`: 144 pass. `runnerd-codex-transport.test.ts -t "cold process admission"`: 4 pass. `cargo test -p paperclip-runner-core --lib admission_budget` passes. - `npx tsc --noEmit -p server` and the runner `tsc -p tsconfig.test.json` pass. ## Risks - **This changes Runner Rust code. The Daytona runner image must be rebuilt after merge** for remote sandboxes to get the 60-second sidecar bound. Until then, a remote Claude `session.open` still fails at 30 seconds in the sidecar (as a definite failure, which already retries). A remote timeout stays operator-owned, because a remote root cannot be proven clean here. - A cold ACPX Claude start that hangs now fails after about 60 seconds instead of 30 seconds. - A timed-out `session.open` may have created an empty provider session. No prompt was sent. The retry quarantines that runner root and asks ACPX for a new session. - Same-run retries of a fresh-attempt Retry run can still fail with `isolated_task_directory_binding_mismatch`. This is RC-35 (workspace re-preparation), which has its own fix in progress. - No schema or API change. The new error code appears on runs and in run error codes. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, file edits, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
e89be6ee83 |
fix(native): re-bootstrap a runner that a graceful drain suspended during bootstrap (#19665)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs start a local `paperclip-runnerd` process. The control plane keeps a durable state root for each session. > - After a restart, recovery gives a run with no provider evidence the `bootstrap_incomplete` disposition. The executor quarantines the old root and starts the bootstrap again, but only when the root proves that no provider work started. > - #19524 made that proof accept a root killed after `run.prepare` and before `session.open` (a hard kill). > - A graceful restart (SIGTERM) leaves different shapes. The drain sends `runner.drain` and `runner.suspend`, and the kill can land just after `session.open` failed or just after it started a fresh session. The proof refused both shapes. The run failed with `runner_state_identity_mismatch: bootstrap_not_proven_incomplete`, although no turn was ever sent. > - This pull request extends the proof to both drain shapes and keeps provider work fail-closed. > - The benefit is that a graceful restart during native bootstrap continues automatically. ## Linked Issues or Issue Description Refs #19524 **What happened?** Restart chaos waves (ACPX Claude, K2 graceful restart about 10 s after the run started) failed with `adapter_failed` and `runner_state_identity_mismatch: bootstrap_not_proven_incomplete`. Startup recovery chose `bootstrap_incomplete`, and the executor proof refused the state root. Two real roots were kept: 1. `session.open` failed (the ACPX sidecar timed out), then `runner.drain` completed and `runner.suspend` completed. There was 1 connection, no committed or acknowledged event, an empty runner outbox, and `nextSourceSeq: 1`. 2. `session.open` completed with a freshly started session (`status: "started"`). The committed events were only `session.started`, `session.capabilities.updated` and `session.goal.snapshot`, and the runner outbox held one `harness.diagnostic` (the sidecar process). No turn command existed, and the sidecar process had exited. The database had not yet recorded the session events, so the classifier saw no provider evidence. **Steps to reproduce** 1. Start an ACPX Claude task on a local runner. 2. Send SIGTERM to the control plane about 10 s after the run starts, before the first turn. 3. Let the control plane restart. **Expected behavior** Recovery starts the bootstrap again, and the task continues without a user action. **Actual behavior** The run fails and the task stops. ## What Changed - `native-bootstrap-proof.ts`, pre-session proof: a `session.open` that is pending or `failed` is now accepted, as are the runner lifecycle commands `runner.drain`, `runner.suspend` and `runner.shutdown` in any state. The controller journal and the runner journal must both agree. Any committed or acknowledged event, any turn command, a completed `session.open`, and any queued runner event still fail closed. - New `runnerdStateIsOpenedSessionBeforeTurn`. It accepts a completed `session.open` only when all of these are true: - the session was freshly started (`status: "started"`, not resumed); - the commands are only `run.prepare`, `session.open` and runner maintenance commands; - every committed event and every event in the runner outbox is a session setup or runner lifecycle event; - `ackedSourceSeq` equals the count of committed events, with no gap; - the runner journal is present; - every recorded provider process ID has exited on this host. - `native-session-executor.ts`: the executor applies the opened-session proof only for a local execution target. Remote runners stay fail-closed for that shape, because their process IDs belong to another host. The remote checkpoint classifier still reads only commands 1 and 2. ## Verification - Red before the fix: "allows a fresh bootstrap after a drain suspended a runner whose session.open failed" and "allows a fresh bootstrap after a drain that landed between session.open and the first turn" both failed with `expected false to be true`. Green after the fix: the `native incomplete-bootstrap evidence` block passes 32/32. - Fail-closed counter-tests: - completed `session.open`; - committed event, acknowledged event, or queued runner event; - turn command, `turn.stop`, or a turn processed by the runner; - `turn.started` or `provider.event`; - resumed session; - compacted events; - no recorded provider process; - missing runner journal; - live provider process; - remote runner. - I ran the new proof read-only against the three real kept roots: the K1 root from #19524, the K2 `session.open` failed root and the K2 opened-session root. All three return `true` for a local runner. The opened-session root returns `false` when the processes are on another host. - `native-session-executor.test.ts` and `native-restart-recovery.test.ts` filtered to bootstrap, restart, remote and drain: 333 passed. `native-runner-restart-recovery.integration.test.ts` and `native-bootstrap-retry.integration.test.ts`: 106 passed. `tsc --noEmit -p server/tsconfig.json` passes. - Live restart chaos replay: see the PR comment. ## Risks - After a `session.open` failure or timeout, the provider sidecar might have created an empty session. No turn command was sent, so no provider work can repeat. - An opened session before any turn is discarded, and a new one is started. The proof requires a fresh `started` session, so no earlier conversation is lost. Resumed sessions stay fail-closed. - The opened-session proof is local only. It requires every recorded provider process to be gone, so a live provider process blocks it. - Out of scope: the Retry offer when the proof fails legitimately (RC-05/06). > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude Code with tool use (shell, file edits, test execution). Default context window. Extended reasoning was available. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
9197abf673 |
fix(native): preserve controller state when remote archiving fails (#19663)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native continuity replacement must preserve the state needed to recover saved work. > - The current replacement moves local controller files before it knows whether the remote archive succeeded. > - A remote failure then leaves the local controller without its recovery files. > - This pull request confirms the remote archive before it moves those files. > - A user turn that never started can use the existing bounded Retry path after verified preservation and cleanup. > - The task retains its saved work and can continue once under current user authority. ## Linked Issues or Issue Description Refs #19638, #19547, #19399 and #19659. The prerequisite #19638 merged as `0819f4ea78dd37fba87341b314be9dac078c1410`. This PR now targets master. A normal dependency integration at `febed950f4` keeps only the six-file archive repair in the PR diff. Fresh master-based CI and review are required before merge. **What happened?** A native continuity replacement moved its local controller journal and provider checkpoint. The following remote archive failed. A later recovery attempt could no longer read the local recovery files. The first-attempt failure also had no normal Retry receipt for the saved user turn. **Expected behavior** A failed or timed-out remote archive retains the local recovery files and starts no replacement provider. A user turn that has not started can request one normal Retry. The existing user, provider-turn, Stop, preservation, cleanup, budget and approval gates remain in force. **Steps to reproduce** 1. Keep a local controller journal and provider checkpoint for a retained remote conversation. 2. Request continuity replacement and fail or time out the remote archive command. 3. Check the local controller files before any later recovery. 4. Settle the native failure under a current user wake and recover the saved workspace. ## What Changed - Confirm remote continuity archiving before moving the local controller state. - Add the exact remote archive failure to the shared bounded user Retry admission from #19638. - Verify failed, timed-out and successful archive ordering with real local files. - Verify ordinary failure settlement and saved-file recovery produce one fresh successor. Keep Stop, system wake and unsettled-provider holds. - Document the file preservation and Retry contract. ## Verification - RED: failed and timed-out remote archives removed the local controller files. Reading those files failed with ENOENT. The saved-user Retry request returned false instead of true. - Initial archive unit verification: eight cases passed, including the three archive outcomes and five existing Cloud archive safety cases. - Initial stacked revision: `9104ee399b`, based on #19638 at `d22d8f9d5ee3dc36550a94d55c1d8376f3b9aab2`. - Final master-based revision: `febed950f4`. All 13 affected cases passed across three Vitest files after dependency integration. They include the seven-mode real PostgreSQL completion case, saved-file successor and negative-hold cases, and real-file archive outcomes. All seven server typecheck layers passed again. Module boundary, whitespace and added-code-comment checks passed. All required CI passed on this exact head. The final snapshot has 59 successful checks, three intended skips, one neutral result, and no pending or failed checks. Current-head Greptile and the independent review are both 5/5, with no open findings. - Final local revision: `9104ee399b`, macOS ARM64, Node 26.4.0, pnpm 9.15.4 and embedded PostgreSQL. The three real-file archive outcome cases passed. The real PostgreSQL completion test passed all seven internal scenarios, including archive failure, prior provider activity, system wake, Stop before and after terminal settlement, and stale runtime promotion. Own Runner/shared/SDK builds and all seven server typecheck layers passed. Nine saved-work continuation and negative-hold cases passed. They verify exported file contents, one user-attributed fresh successor, idempotent hooks, no second automatic chain, and first-use manual Retry for held cases. - `node scripts/check-module-boundaries.mjs`, whitespace checks and the added-code-comment audit passed. - Focused commands from `server/`: Vitest with `-t 'retains the local controller checkpoint until remote continuity archiving succeeds'`; `-t 'replays saved user intent through normal Retry'`; and `-t 'continues a user turn once|does not continue automatically'`. PostgreSQL cases used a fresh local cluster and a 120-second setup hook budget. Application timeouts were unchanged. - The initial stacked head passed all test, build, typecheck and E2E jobs. Greptile and the independent review both scored it 5/5 with no findings. These results do not replace the fresh master-based qualification now running. ## Risks - A remote command can fail after it changed remote state. This change retains local evidence in that case. It does not claim a distributed atomic transaction. - The exact archive error can request one normal Retry only under the existing saved-user admission. The normal path must prove stop, saved files and cleanup before a successor starts. - The protected launch-argument rejection remains enforced. Its exact hosted argument difference was not available through ordinary access. #19659 separately addresses a saved-thread feature-policy argument difference. This PR preserves local archive evidence and does not claim direct hosted verification of that primary cause. ## Model Used OpenAI GPT-6 family through Codex, with reasoning, tool use and code execution. The model is inherited; no override was used. The exact serving model ID and context-window size are not exposed by this runtime. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
374a918d67 |
fix(runner): retain qualified OpenCode artifacts for cold recovery (#19622)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native recovery must keep admitted provider state and useful workspace files. > - OpenCode binds its saved session to exact executable paths and hashes. > - A replacement image can remove those files even when session data is saved. > - Retaining a manifest alone cannot restore the missing launch files. > - This pull request saves the qualified files before provider work and restores the same owned paths. > - The benefit is cold continuity without relaxing the Runner guard or replaying a prior turn. ## Linked Issues or Issue Description Refs #19558. That fix retains the selected pack for ordinary continuation. This change adds durable file retention for a cold allocation. Related #19487 adjusts the admission budget; it does not retain launch files. A search of open and closed OpenCode cold/durable artifact PRs found no duplicate. **What happened?** A saved OpenCode session could outlive the image that supplied its Node executable, proxy, and native executable. A new image supplied different bytes. The Runner then correctly refused the changed launch profile. The saved harness did not contain the old files. **Expected behavior** Core must save qualified launch files before the first turn. It must restore those exact files under the same owned account and absolute paths. Unknown ownership, missing files, and changed hashes must still stop recovery. **Steps to reproduce** Start an admitted remote OpenCode session. Save its workspace, encrypted checkpoint, and harness data. Destroy the original allocation and controller files. Restore into a new allocation with an image whose qualified proxy has different bytes. Resume the same session. The new opt-in local qualification command automates this path with actual OpenCode and a local synthetic model. ## What Changed - Store company-bound, encrypted launch objects and canonical run/backup references. Generate migration 0438 through the normal database workflow. - Stage qualified Node, proxy, and native OpenCode files under a private content-addressed OS-account home. Record the account UID and exact launch paths. Preserve the Rust profile guard. - Gate first provider work on durable capture under the current native claim. Bound Stop/deadline waits while the actual late transfer keeps its worker and scratch ownership until settlement. Restore cold caches with verified staging and atomic rename. Keep live reattachment and historical image-rooted sessions on their existing paths. - Reuse bounded canonical expiration owners. Atomically fence old checkpoint authority with a strict company/run-scoped retirement marker before external deletion. Keep its indexed cleanup anchor across failed deletes and restart. Preserve earlier resume handles and reject later header or source-consumer writes. Add no idle scan or network call inside a transaction. - Add storage, race, executor, filesystem, and actual Rust resume regressions. Add the opt-in local Linux qualification command and operating limits in `doc/opencode-cold-recovery.md`. ## Verification - RED before repair: real PostgreSQL/AES cold restore lacked the launch files; normal executor preparation chose the new image pack. Separate regressions exposed lost upload receipts, missing cache repair, last-owner retirement, an account-UID mismatch, and a resume-handle race during external deletion. Each expected failure was captured before its repair. - Local qualification before the final master integration: 1,169 executor tests, 233 storage/archive/admission/expiration tests, all seven server typecheck layers, migration safety, and the actual Rust cold-resume test passed. - The maintained Linux command passed with two cached immutable Paperclip images whose proxy hashes differ. It removed the original account/controller trees, restored the production published cwd under a new lease, resumed the same actual OpenCode session, kept a useful file and prior model context, and made one new synthetic model request. No external model call or network was available in either container. - The object-store fixture uses real encrypted StorageService streams with an independent owned local directory. It is not hosted S3 acceptance or a complete task HTTP/actor workflow. Historical tasks still require normal effects reconciliation and recovery after deployment. - Review regressions reproduced partial-cache publication, resume-versus-deletion, fresh-writer and source-consumer resurrection, and unbounded Stop/deadline waits before each repair. They now pass through real filesystem, encrypted storage, PostgreSQL, and SDK boundaries. File validation is split from database ownership (384/831 source lines). - Clean runtime head `aadab7acc69ff959678030db46ffe2dcfbefe28f`: the maintained actual OpenCode cold qualification passed with `dirty:false`. It preserved the provider session and recorded launch paths, restored useful workspace content and prior model context, and made exactly one resumed synthetic model request. The 314,560,000-byte launch capture took 23.118 seconds; the later harness was 833,536 bytes. Subsequent repairs change backup-reference retirement and governed continuity-cache classification; capture, restore, launch, and published-workspace inputs are unchanged, so that runtime receipt remains scoped to those paths. The continuity-break consumer is verified separately. - Retention repair `c14d386cecf7123257918ba6a8d47307df2da397`: 68/68 affected real PostgreSQL/AES storage tests and 15/15 focused executor authority/admission cases pass. All seven server typecheck layers pass (source layers and unchanged test shards, with the final 590-file test layer repeated after fixture edits). All review threads have evidence-backed resolutions. Its subsequent CI caught the separate lifecycle-classification gap described below. The refreshed Paperclip review reported paused work, not a completed new review. - Final head `4d1cd3e3309e328b11abc85adc6d78c6d0c7f55d`: all 19 focused root-entry/executor cases and all three source declaration layers pass. [Greptile reviewed this exact head at 5/5](https://github.com/paperclipai/paperclip/pull/19622#issuecomment-6695984874). Final-head [CI verification](https://github.com/paperclipai/paperclip/actions/runs/42308908167/job/128214100339) passes: 60 successful checks, five conditional skips, and the explicitly neutral paused Paperclip review. No check is pending or failing. - CI also caught the launch cache missing from the session-root lifecycle contract. Both the writer scan and a real governed-replacement consumer failed locally before repair: old launch bytes remained in the active root. The cache now archives with the prior session state; the consumer verifies exact bytes are preserved in the archive and absent from the active root. - The latest review found that backup retirement could delete an independent run pin while a selected session retained the checkpoint. A real PostgreSQL/AES regression failed before the repair. Backup-only ownership now uses a NULL-safe exact backup predicate; the run pin belongs to fenced checkpoint expiration. The regression verifies checkpoint decryption and launch-file restoration after pending-backup cleanup, followed by deletion only after removing the final resume handle. - An executor first-session case checks new-session durable capture through the real capture helper and receipt propagation into the actual harness manifest at the backup boundary. Real storage cases verify the run-profile receipt and canonical backup/run pins. - Sequential closure reuse and simultaneous first admission are different scopes. A deterministic two-uploader test measures both independent uploads, one verified ready winner, loser-only deletion, and usable restoration for both runs. Each capture is bounded to 512 MiB and five minutes; up to three pending receipts per run are retained. There is no claim of global single-flight capture or a company-wide egress bound. - Earlier optional warm-ownership verification encountered missing local binaries. After exact-source setup, all nine ownership assertions passed; two other suites encountered local PostgreSQL setup/teardown contention, so those interrupted attempts are not passes. Required current-head CI supplies the broad final coverage. No deployment or hosted historical recovery is claimed. ## Risks - This adds two tables and a forward migration. The file object is bounded to 512 MiB. Capture adds one initial transfer; later harness archives exclude those large files. Company-scoped ready objects are shared and retained until the last canonical owner retires. Concurrent initial admissions can duplicate bounded transfers; each keeps independent ownership and cancellation rather than waiting on another controller. A continuity break retains the controller cache with its prior state until normal session-root cleanup, adding up to one bounded launch closure per retired state. - Cold continuity requires the same qualified Linux platform, supported runtime dependencies, owned account UID/home, and published absolute cwd. A mismatch fails closed. No alias or Rust guard change is included. - Existing image-rooted profiles have no retained file object. This change cannot backfill missing historical binaries. A fresh governed session must reconcile prior effects first. - Live reattachment does not read artifact objects. Storage or provenance failures before a new turn hold admission. They do not grant new execution, model, tool, or replay authority. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, repository tools, code execution, and local browser tools. The exact served model ID and context window are not exposed in this session. No claim is made for unavailable model metadata. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
7941f6b051 |
fix(runner): show OpenRouter's invalid model ID as a model rejection (#19548)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs use the Paperclip runner with drivers such as OpenCode
> - When a provider rejects the selected model, the product shows "Model
unavailable" and offers a model choice
> - OpenCode reports provider HTTP errors as `APIError` with the message
under `data`, not at the top level
> - The runner then reported "Provider turn failed", so an invalid model
ended as a generic adapter failure
> - This pull request classifies OpenRouter's invalid model ID with a
closed provider-failure category, and makes the server keep that
classification
> - The benefit is that a person sees "Model unavailable" and the real
provider message, and can choose another model
## Linked Issues or Issue Description
**What happened?**
A chat evaluation selected a model ID that OpenRouter does not know.
OpenCode reported `turn.failed` with `error: { name: "APIError", data: {
message: "<model> is not a valid model ID", statusCode: 400,
isRetryable: false } }`. The run was saved with `errorCode:
adapter_failed`. The task chat showed "The OpenCode run failed before it
completed." and no "Model unavailable" marker. The provider notice said
only "OpenCode session failed.".
**Expected behavior**
A non-retryable rejection of the selected model is saved as
`native_provider_model_rejected`, so the product shows "Model
unavailable" and the model choice. The notice shows the provider's
message.
**Steps to reproduce**
1. Configure an OpenCode agent with an OpenRouter model ID that does not
exist.
2. Send the agent a message.
3. Before this change, the run fails as `adapter_failed` and the notice
says "OpenCode session failed.".
## What Changed
- `drivers/opencode/provider-failure.ts`: a non-retryable OpenCode
provider error whose message says the model is not valid (`not a valid
model`, `model_not_found`, model not found or not supported) gets the
closed category `model_unavailable`. A provider that allows a retry
keeps the existing category.
- `native-session-runtime.ts`: a non-recoverable failure with
`model_unavailable` is a model rejection. The runtime reads the closed
category, not provider text.
- `native-session-executor.ts`: a non-recoverable
`NativeProviderTerminalFailure` that the runner marked as a model
rejection is saved as `native_provider_model_rejected`. Before, every
such terminal was saved as `native_provider_terminal_failed` before the
marker was read.
- `drivers/opencode/opencode-server-driver.ts`: the provider notice
summary shows the provider message from `data` when there is no
top-level message. Provider text is not used as the turn failure
message, so it does not reach `run.error` or Sentry.
- Tests: provider-failure mapping (retryable and non-retryable), runtime
classification by category, driver notice summary with no failure
message, and an executor test for the saved code and terminal
disposition.
## Verification
- `npx vitest run src/native-session-runtime.test.ts
src/drivers/provider-failure.test.ts src/drivers/opencode` in
`packages/paperclip-runner`: 16 files, 412 tests passed.
- `npx vitest run
src/services/native-runtime/native-session-executor.test.ts` in
`server`: 1,152 tests passed.
- Negative controls: reverting the provider-failure mapping, the runtime
check, or the executor branch each makes its new test fail.
## Risks
- Low risk. Classification uses a closed category and a runner-set
marker, not provider text.
- The model check on the provider message is text-based inside the
OpenCode mapper. A provider that uses "not a valid model" for another
reason with a non-retryable error will show "Model unavailable".
- Provider text appears in the notice summary, as provider notices
already did for top-level messages.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
ee6c101854 |
Reject failover backups behind the accepted event cursor (#19613)
A verified failover backup can still trail events already accepted by the controller. Compare the backup produced cursor with the registered controller state, or the supported local journal before registration, and reject stale files before remote sync. Keep the permanent replay-conflict classification ahead of generic Runner errors.
Validation: 1,192 native session and execution-driver tests and 10 workflow tests pass locally, together with the maintained low-memory Runner test project typecheck and fresh server declarations. All applicable checks on
|
||
|
Nicky LeachandClaude Opus 5.5
|
a7d3d4b78c |
fix(runner): rotate local and external authority through one resumable path (#19547)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The Paperclip Runner moves a native session from one run to the next
with an authority rotation, for Codex and for every ACPX provider
> - A rotation archives the prior run's controller state under
`authority-epochs/` and archives the Runner state, locally or on an
external Runner host
> - If the run fails after the rotation but before the new Runner
starts, retry preparation restores the prior run's controller checkpoint
into the session root
> - The retry then finds an active `control-plane` next to the archive
marker and fails with `native_runner_authority_archive_conflict`. Every
later run for the session fails the same way
> - The local and external rotations were two copies of the same steps,
with different retry rules, and the sandbox archive script had a third
set of rules
> - This pull request gives every provider and Runner placement one
resumable rotation path, with the same retry rules for controller state
and Runner state
> - The benefit is that one failed startup no longer makes a native
session unusable
## Linked Issues or Issue Description
No public issue exists. Bug description:
**What happened?**
A warm Codex session on an external (plugin) Runner failed its first
attempt after the authority rotation finished. The two automatic retries
both failed with `native_session_recovery_failed: Error:
native_runner_authority_archive_conflict`, and the run stopped. On disk,
the session root had
`authority-epochs/epoch-<key>/control-plane/control-plane-state.json`
and an active `control-plane/control-plane-state.json` with the same
identity (in this case, also the same sha256). The remote Launcher had
already archived the Runner state under the same key.
**Expected behavior**
A retry resumes the interrupted rotation. The rotation already supports
this when only the archive marker exists.
**Steps to reproduce**
1. Run a native Codex or ACPX session on an external Runner.
2. Start a new run for that session. Make the new Runner fail to start
after the rotation finishes.
3. Let the run retry. Retry preparation (`restoreWarmRunnerdController`
/ `restorePerTurnRunnerdController` →
`restoreNativeControllerProjections`) restores the prior run's
controller checkpoint into the session root.
4. The retry fails with `native_runner_authority_archive_conflict`. All
later runs for the session also fail.
The new `authority rotation after the prior controller state is
restored` tests reproduce steps 2–4 directly.
**Paperclip version or commit**
`master` at
|