PaperClipAI

0 Fork 0

mirror of https://github.com/paperclipai/paperclip.git synced 2026-11-12 14:19:00 +01:00

Author	SHA1	Message	Date
DottaandClaude Opus 5.5	f755361ffe	fix(recovery): deliver a steered message the restart lost before the model read it (#19962 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A user can steer a running Codex or Pi turn: a comment goes into the live turn instead of waiting for the next run > - The provider acknowledges the steer at once, but the model reads it only at its next step, after the in-flight tool call > - A control-plane restart in that window kills the provider with the message in memory. Restart recovery starts a continuation turn that does not carry the message. The message is lost and no error shows > - The restart chaos harness showed this in 6 of 6 steered runs (Codex, K1, K2 and K6, 2 samples each): steer acknowledged, run succeeded, no reply to the steered message > - This pull request makes restart recovery deliver every acknowledged steer that the model did not read, exactly once > - The benefit is that manual Steer no longer loses messages on restart, and the planned automatic steering can ship on top of it ## Linked Issues or Issue Description What happened? A comment steered into a running Codex turn got no reply when a control-plane restart (SIGKILL, SIGTERM or crash loop) happened during the agent's tool call. The run succeeded, and the steered comment had no effect. Steps to reproduce 1. Start a Codex task that runs a 40-second command. 2. During the command, post a comment and click Steer. The provider acknowledges it. 3. Restart the control plane before the command ends. 4. Recovery resumes the turn with a continuation. The agent finishes the task and ignores the comment. Expected behavior The agent reads and handles the steered comment once, after recovery. Root cause The provider keeps acknowledged steering input in memory until its next model step. The restart continuation (`paperclip.native-continuation.v1`) only told the model to continue the current request, so the acknowledged input was gone. ## What Changed - `unreadSteeredMessageIds` (runner package): derives acknowledged-but-unread steers from run-log events in order. A `steering_acknowledgement` item adds the message id (once per id). A `userMessage` item in the same turn reads the oldest unread steer. Codex records each steered input as one `userMessage` item when it enters the conversation. A `userMessage` item in a later turn reads all earlier steers, because the continuation carries them. `turn.completed` reads all steers. - `readUnreadSteeredMessages` (server): reads those facts from `heartbeat_run_events` (only transport-validated events with a `sourceEventId`). It also reads the comment or interaction text for the same company and issue. The run log is the durable record of both "acknowledged" and "read", so no new column is needed. - The restart continuation now includes `steeredMessages` and one instruction to handle them, oldest first. The runtime reads them through a lazy `getUnreadSteeredMessages` option, only when a restart continuation starts. Native recovery wires it for every run. - `queuedInteractionSteeringBody` is extracted, so recovery rebuilds a steered interaction response with the same text the steer sent. - Fake Codex gets `--consume-steer`, which emits the `userMessage` item for an accepted steer, like real Codex. ## Verification - Red/green live test with real runnerd and fake Codex (`runnerd-restart-continuation.test.ts`, "delivers a steered message the restart lost exactly once"): start a held turn, steer it, SIGKILL runner and provider, restore and recover. - "unread": the continuation carries the message exactly once. Without the runtime change this test fails. - "read" (`--consume-steer`): the continuation does not carry it again. - Both: after the continuation completes, a later restart has nothing to deliver. - `steering-delivery.test.ts`: ordering, dedupe, initial-input, later-turn and completed-turn rules. - `unread-steered-messages.test.ts` (embedded Postgres): ignores events without a source id and messages from other issues. It reads interaction text, ignores model-output items as read evidence, and returns nothing after the continuation input. - `pnpm vitest run src/steering-delivery.test.ts src/live/runnerd-restart-continuation.test.ts` (runner) and `pnpm vitest run src/services/native-runtime/unread-steered-messages.test.ts` (server): pass on the rebased revision. - `issue-queued-comments-routes.test.ts` and `native-session-runtime.test.ts -t restart`: pass. `native-session-resume.test.ts` "archives a damaged prior epoch…" fails locally on plain `origin/master` too (local runnerd binary); CI runs it with a fresh build. - Restart chaos (Codex steer mode, K1/K2/K6) runs after this lands in the harness slot. Before: 0 of 6 steered messages answered. ## Risks - Duplicate delivery: a provider that records steered input without a `userMessage` item, or a `userMessage` event lost in the crash window before persistence, makes recovery send the message again. The model then sees it twice. That is safer than losing it, and the instruction asks the model to handle each one once. - Pi (ACPX) steering has no `userMessage` read evidence today, so a Pi steer that the model already read before a restart may arrive again in the continuation. - Only the restart continuation carries the messages. Other recovery paths start after a completed turn, which already read every steer. - No schema or migration change. A session saved by an earlier release resumes normally; its run log already holds the acknowledgement events. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file editing, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 22:22:52 -06:00
DottaandClaude Opus 5.5	bf3a067076	feat(recovery): settle the finalizer and cleanup leases on the shared ownership path (#19950 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs keep their ownership in `native_run_finalizations`, and the shared ownership reconciler now settles proven-gone owners (Phase 2 of the durable ownership plan) > - The finalizer lease (`native-finalizer:`, 5 min) and the retained-session cleanup lease (`native-cleanup:`, 60 s) recorded only a random token. They named no holder process > - A server killed while it finalized or cleaned up kept the run waiting for the whole lease. The lease was released by age alone. A graceful shutdown also keeps an in-progress finalizer lease for the next boot to wait out > - This pull request records the holder process with each claim and moves both kinds onto the shared prover and a compare-and-swap settle > - The benefit is that a dead holder's run continues in the first pass after boot, and a live or unprovable holder is never released ## Linked Issues or Issue Description Refs #19933 (the settle hook and the first moved kind, merged). What happened? A finalizer or cleanup lease whose server process died stayed held until the lease expired (5 min or 60 s). The run stayed `running` and its follow-up work waited. What did you expect? The next boot proves the dead holder gone and frees the lease at once. A live holder, or one that cannot be proven gone, keeps its lease. Steps to reproduce 1. Accept a native result and let the finalizer claim it. 2. Kill the server (SIGKILL) before the finalizer commits. 3. Start the server. The run stays `running` for up to 5 minutes. ## What Changed - `native_run_finalizations.lease_holder` (jsonb, migration 0446): `{token, bootId, pid, hostHash, processStartedAt}` of the process that took a token-only lease. It names the holder only while `token` equals `lease_owner`, so renewals keep it and a later lease never inherits it. The executor's `controller_*` columns are not changed. - The finalizer claim (`native-run-finalizer.ts`) and the cleanup claim (`native-session-executor.ts`) write it in the same UPDATE as the lease, from `currentControllerProcessStamp()` (no I/O). - New `native-token-lease-settlement.ts`: compare-and-swap settle that expires exactly the proven-gone holder's outstanding lease, after it notifies the finalization or cleanup work that the expiry frees. - Both claims also wake the ownership worker queue in their transaction, so a quiet worker watches the new holder (Greptile round 1). - `ownership-registry.ts`: `native_finalizer_lease` and `native_cleanup_lease` register `settle`. Their owner is the holder process; rows without a holder keep the token fence. Listings cover outstanding leases only. - `heartbeat/recovery.ts`: restart recovery settles both kinds before it finalizes and schedules cleanup. A failure is logged and the reconciler pass retries it. - Two sparse partial indexes on `lease_expires_at` for the two listings, so the settle watch re-check stays indexed. Bespoke code deleted: none. These kinds had no liveness check; their only release was lease age (the takeover in the claim), which stays as the fence for unprovable holders. ## Verification - `cd server && npx vitest run src/__tests__/heartbeat-native-restart-suspension.test.ts -t "finalizer\|cleanup\|holder\|exact lease"`: 12 passed (rerun after the rebase on master). The crash-point test also asserts the ownership wake for its run; it fails without the signal. - `cd server && npx vitest run src/__tests__/ownership-reconciliation-report.test.ts`: 17 passed. - Red on the base branch (server code at `bea3bfb20b`, tests and schema from this branch): 7 new tests fail, 5 pass (the 5 are existing tests). Red: the crash-point restart test, the two reconciler settle tests, the three live/unprovable counter-cases (owner was a token fence), and the exact-lease CAS test. The tests were written after the code. - New tests: - crash point: a finalizer claim was written and its holder killed before commit; the next restart finalizes the run, and this boot's claim stamps itself as holder; - a dead finalizer or cleanup holder is settled by a recording pass, never by a read-only pass (`record: false`); the accepted result and phase are untouched; - live counter-cases: a live earlier-boot PID (`pid_alive`), the current boot (`current_process`), a dead PID in another PID space (`different_process_space`, unprovable) keep the lease through a pass and a restart; - a holder written for another token, and a holderless row of the previous release, stay token fences, also with `PAPERCLIP_EPHEMERAL_SERVER=true`; - settle refuses a lease taken over after the proof (`owner_changed_since_proof`). - Not run locally: typecheck (CI runs it) and the wide suites. ## Risks - Previous-release story (rules/upgrade-safe-run-state.mdc): rows saved by the previous release have no `lease_holder`. They stay token fences and the existing claim takes them over at expiry. Neither lease outlives 5 min, so this covers at most 5 min after the upgrade. A rollback ignores the new column. - Unprovable holders (other host or PID space, unknown space): no operator action. The lease is short and self-expiring; at expiry the next claim takes it over and the old holder's next token compare-and-swap fails. That fence is the handoff. A different host is never proof. - The settle only expires a lease. Finalization and cleanup still run under their own checks (cleanup keeps its maintenance-history proofs). Settling ownership does not settle the accepted result, workspace preservation, or accounting. - Migration adds a nullable column and two partial indexes on a sparse predicate. Low risk. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, local test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 22:16:34 -06:00
DottaandClaude Opus 5.5	a4a3e342ec	fix(recovery): charge a restart attempt only when the provider accepted a turn and made no progress (#19948 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A run has an execution recovery budget: three provider attempts for native runs, and two failure retries for legacy runs > - In the restart chaos campaign, control-plane restarts used up that budget for runs that were healthy (RC-22, 22b, 22c, 40 and 60) > - #19720 and #19747 fixed two variants with watermark heuristics, and legacy interrupted preparation still spent the failure budget > - The durable-ownership plan (section F, invariant 5) asks for one rule: charge an attempt only when a recorded outcome proves that provider work started and no progress followed > - This pull request adds that rule and moves the native restart charge and the legacy preparation retry onto it > - The benefit is that a restart never spends budget for work the provider never started, while real failure loops and crash loops stay bounded ## Linked Issues or Issue Description Refs #19720, #19747, #19856 What happened? A control-plane restart charges a provider attempt by heuristics. The first crash restart always charges. A later restart charges unless the previous resume had no start evidence (#19747) or completed work (#19720). A runner that spawned but whose provider never accepted a turn counted as started, so a K6 crash loop could still spend the budget. Legacy runs whose preparation a restart interrupted (`providerWorkStarted: false`) spent `failureRetries`, the same counter as real failures (`execution-recovery-attempt.ts`). Expected behavior An attempt is charged only when the provider accepted a turn (`turn.accepted` or `turn.started`) and no accepted turn then completed work. A startup before spawn, a runner that spawned but was never admitted, and a `turn.start` the provider did not accept are not charged. The generation-12 cap still bounds uncharged restarts. Steps to reproduce Start a native run at its last attempt. Kill the controller after the runner spawns but before the provider accepts a turn. Before this change, the next restart claim blocks the run with `execution_recovery_budget_exhausted`. ## What Changed - New `server/src/services/native-runtime/provider-attempt-outcome.ts`: the one rule. `readProviderAttemptOutcome()` returns `provider_not_started`, `provider_started_without_progress` or `provider_progressed` from run events. `providerAttemptChargesBudget()` charges only the second. `providerAttemptWindowStart()` starts the attempt window at the latest `native.process_start_requested`, so earlier work never counts for a later attempt. `PROVIDER_ATTEMPT_CREDIT_GENERATION_LIMIT` (12) is the shared cap. - `restartResumeKeepsAttempt()` (`native-restart-recovery.ts`) now applies that rule to the attempt the restart interrupted, including the first restart. It keeps the unstarted-resume carry from #19747, the reattachment rule (an adopted provider counts as started), the drain exemption from #19625, and the generation-12 cap. A history entry from an older release without a recorded position still charges. - Legacy runs: `interrupted_preparation` is now an uncharged retry lane (`execution-recovery-attempt.ts`). It keeps its own counter, bounded at the same cap (12), and never changes `failureRetries`. `retries.ts` uses the longest transient delay after the second lane attempt. - `doc/execution-semantics.md` states the rule. - Accounting has its own "provider work started" readers (inventory Summary finding 5). Phase 3 owns those; this PR exposes the rule for them to adopt and does not change them. ## Verification - `native-runner-restart-recovery.integration.test.ts` (embedded Postgres, real `claimNativeRestartRecoveries`): 129 passed. New cases, red on master: - not charged at the last attempt for a claimed remote bootstrap, a remote bootstrap retry, and a claimed unlaunched remote run; - not charged when the runner spawned but was never admitted, and when the provider did not accept `turn.start`; - not charged on the first restart when the original attempt already made progress; - a normal retry that spawned but was never admitted is not charged, and earlier work still does not count for it. - Counter-cases, green on master and with the change: an accepted `turn.accepted` or `turn.started` with no progress is charged; the generation cap blocks at the last attempt; crash loops with and without progress stay bounded; the #19720 and #19747 cases still pass. - `execution-recovery-attempt.test.ts`: new lane tests (red on master). - `workspace-recovery-preparation-dispatch.test.ts` (25 passed), `heartbeat-process-recovery.test.ts` (interrupted-run cases), `run-dispatch/adapters/postgres.test.ts` (interrupted cases). - `server` typecheck (`scripts/typecheck-layers.mjs`, low-memory mode): no errors. - Not run: chaos waves (the coordinator schedules them on this host), and the full server suite (CI covers it). ## Risks - Behavior change: the first crash restart no longer charges when no accepted turn is recorded. If a crash lands before the controller projects the provider's `turn.accepted`, that attempt is not charged. This fails toward more retries, and the generation-12 cap bounds it. - A legacy preparation crash loop now gets up to 12 retries instead of 2. Each retry still passes the normal issue gates and keeps the transient delay. - The new queries use the `(run_id, seq)` index, once per restart claim. Nothing new runs before `turn.start`. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test execution). Extended reasoning enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 21:20:05 -06:00
DottaandClaude Opus 5.5	18644a3263	feat(recovery): instrument 20 durable write boundaries with crash points (#19938 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A control-plane restart between two durable writes leaves an in-progress record owned by a dead process > - #19937 added `crashPoint(name)` and a registry of 20 named boundaries from the Phase 0 inventory > - A name without a call site cannot fire, so the crash-point suite cannot test it > - This pull request puts one call at each registered boundary and adds a test that keeps the registry and the call sites in step > - The benefit is that the on-demand suite (next pull request) can kill the control plane at every known write-ahead gap ## Linked Issues or Issue Description Refs #19831 (plan, Phase 5). Refs #19846 (inventory, "Write-ahead coverage"). Stacked on #19937 (helper and registry). Refs #19861 (ownable kind ids). Problem The registered crash points have no call sites. Expected behavior Each registered point is called at its gap: after the first durable write or outside effect returns, before the next one starts. Disarmed (always in production) each call is one comparison. Scope Call sites and a coverage test only. The suite runner and CI workflow follow. ## What Changed - Server call sites (`@paperclipai/shared/crash-points`): - `sandbox-creation-operations.ts`: around the provider `create` call (intent → provider → lease). - `native-session-executor.ts`: after `native.process_start_requested`, before spawn. - `paperclip-control-plane-port.ts`: after the result-accept transaction, before finalization starts (only when a result was inserted). - `native-run-finalizer.ts`: after `commitNativeStatusDecision`, before the run projection. - `workspace-operations.ts`: `workspace_finalize` only — after the running row, and after `settle()` before the done row. - `environment-runtime.ts`: provider release → lease row; teardown claim → destroy; reusable destroy → lease release; stop-and-retain claim → stop. - `heartbeat.ts`: lease release → runtime services; budget reservation hold → dispatch. - `heartbeat/queue.ts`: run claimed `running` → wake request `claimed`. - `run-cost-accounting.ts`, `usage-receipts.ts`: accounting commit → spool unlink; receipt spool → DB row → unlink. - `packages/paperclip-runner/src/crash-point.ts`: a small copy of the helper for controller code in this package, which does not depend on `@paperclipai/shared` (adding that dependency would change the runner content ID). It arms only `runner.*` names. runnerd never sees the variable: its environment is an allowlist. - Runner call sites: after the runnerd spawn, before its identity is published (both spawn paths); `turn.start` result received → controller journal saved → next send. - `server/src/__tests__/crash-point-call-sites.test.ts`: every registered point has a call site, every call names a registered point, and every `kinds` entry is an ownable kind id of the Phase 1 registry. ## Verification - `cd server && npx vitest run src/__tests__/crash-point-call-sites.test.ts` → 3 passed. - `pnpm --filter @paperclipai/server typecheck` → clean. `pnpm --filter @paperclipai/paperclip-runner exec tsc --noEmit -p .` → clean. - Behavior with nothing armed is unchanged: each call returns at its first comparison (measured in #19937: about 3.7 ns). ## Risks - Low risk. No call changes control flow unless `PAPERCLIP_CRASH_POINT` names it and `NODE_ENV` is not `production`. - Points on the path before `turn.start` (process start, runner spawn) cost nanoseconds, within plan invariant 8. - `usage-receipts.ts`: the persist-then-unlink one-liner became a block with the same statements in the same order. - The runner package change gives the runner a new content ID, as any runner source change does. No persisted run state changes (rules/upgrade-safe-run-state.mdc). ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits) and a read-only exploration subagent that mapped the boundaries. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 20:56:33 -06:00
Nicky Leach	94718ce168	fix(native): continue on a replacement when a retained Runner retires before claim (#19897 )	2026-11-07 14:51:32 -08:00
DottaandClaude Opus 5.5	99529f9d71	feat(observability): local speed budgets and a run-log budget check (#19863 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When someone messages an agent, the reply must start as fast as possible, and restart-safety work must not slow it > - The durable ownership plan makes this invariant 8 ("fast turns") and asks Phase 1 to turn the Phase 0 baseline into p50/p95 budgets per adapter > - Without budgets, a regression in time to first token, agent boot or restart recovery shows up only when someone reruns the baseline by hand > - This pull request records the budgets and adds a per-run check that writes a run-log row when a run exceeds its p95 budget > - The benefit is that every later phase can show "no regression against the budgets" from the run log, without new waits on the turn path ## Linked Issues or Issue Description Refs #19831 (plan, "Performance constraint", rule 4, and invariant 8). Refs #19846 (the Phase 0 baseline these budgets come from). Stacked on #19861. Problem The plan requires p50/p95 budgets per adapter for TTFT, agent boot and restart recovery latency, and a check against them. Neither exists. Expected behavior Budgets are recorded in the plan and in code. A run that exceeds a p95 budget gets one run-log row that names the metric, the measurement and the budget. Impact Phase 2–5 exit criteria can cite budget breaches from chaos waves and staging. The check adds no wait before `turn.start`. ## What Changed - New `server/src/services/native-runtime/native-speed-budgets.ts`: the local budget table (codex, claude, opencode) for TTFT and agent boot (cold and warm) and restart recovery latency. Each budget is 1.2× the baseline, rounded up to 0.5 s. `evaluateSpeedBudget` judges one run against p95. An unknown start kind uses the looser budget. Remote runs and adapters without a baseline are not judged. - `native-run-trace.ts`: the trace takes an optional `budget` subject. It checks `task.time_to_first_token`, agent boot (end of `heartbeat.queue` to the end of the first `runner.turn.submit` or `runner.session.resume`) and restart recovery once each. It writes `run.performance.budget_exceeded` through `defer`, so the write is detached from the event path. `finish` now drains diagnostics that defer more work. - `native-session-executor.ts`: passes the adapter and environment to the trace, labels cold or warm at the existing `provider_session` reuse receipts, and checks restart recovery at a recovered run's first provider event, measured from the startup recovery `ready` time. - Docs: budget tables in `doc/plans/2026-11-07-durable-ownership-and-reconciliation.md` ("Speed budgets (Phase 1)"), and the new event in `doc/run-log-events.md`. ## Verification - `npx vitest run src/services/native-runtime/native-speed-budgets.test.ts src/services/native-runtime/native-run-trace.test.ts` (in `server/`): 34 tests pass. They cover: - every budget has p50 ≤ p95; - the provider-to-adapter mapping; - p95 judgement, and the looser budget when the start kind is unknown; - one breach for a slow boot, and none for a second submit or a fast TTFT; - a deferred TTFT breach is written before the trace closes; - restart recovery is checked once; - remote runs and runs with no budget subject write nothing. - `npx tsc --noEmit -p server`: no errors. ## Risks - Low risk. This is a run-log change only (AGENTS.md §5.7, no extra review). The OTLP and Telemetry paths are unchanged. - The baseline samples are small (2–12 per cell) on a shared host. Budgets carry 20% headroom, and the codex restart budget (8.0 s p95) is tight. Expect some breach rows on a loaded machine. They are diagnostics and change no behavior. - Restart-recovery budgets use run 4 of the baseline, as merged in #19846. - Restart recovery is measured from the startup recovery `ready` time to the first provider event, which matches the baseline definition. - Previous-release story: no persisted run state changes. The new event is additive. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 15:27:42 -06:00
DottaandClaude Opus 5.5	76994c5027	fix(native): bound failover backup restores by inactivity, not total time (#19906 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native per-turn session that continues an earlier conversation first restores its failover backup from durable storage. This happens before the coordinator claim, so no provider has started. > - That restore ran under a fixed 300 s total deadline (`AbortSignal.timeout(300_000)`). The deadline covers the download and the local decrypt, digest, extraction and provider-artifact steps. > - On staging, restore time for resumed sessions grew during the day. It reached 300 s, and then every resumed run of one review agent failed with the generic "The operation was aborted". Fresh sessions were not affected. > - A failure at this point also enters the saved-work reconciliation path, so the user request stopped. > - This pull request bounds each durable read by inactivity instead of total time. A slow transfer continues while bytes arrive. A read that stops sending bytes fails with a named code. > - The benefit is that resumed sessions no longer fail only because their backup is large or the store is slow. ## Linked Issues or Issue Description Refs #19894 (adds the same idle-deadline rule for workspace snapshot reads; this PR covers the separate native failover-backup reads). What happened? On staging, a PR-review agent had 17 runs fail in about 80 minutes. Each failure had the message "The operation was aborted" and the code `native_continuation_requires_reconciliation`. Every failing run resumed an existing session (`sessionIdBefore` set, per-turn lifecycle). Every fresh-session run in the same window succeeded. In each failing run, the error came 304–309 s after the `environment.startup` span, and no `native.coordinator.claim` span was recorded. The run result had `executionRecovery: { kind: "bootstrap", providerWorkStarted: false }`. Earlier resumed runs of the same agent show the time from `environment.startup` to `native.coordinator.claim` growing from about 30 s to about 200 s before the failures started. The only fixed 300 s deadline on that path is the failover-backup restore in `restoreDurableHarnessForExecution`. Expected behavior A resumed session restores its backup when the store keeps sending bytes, even when the total time is more than five minutes. A read that stalls fails with a named, diagnosable code. A failed restore publishes no partial backup and keeps the saved backup. Steps to reproduce 1. Persist a native failover backup with provider artifacts. 2. Restore it through a storage reader that sends bytes slowly, so the full restore takes longer than the deadline. 3. Before this change, the restore fails with "The operation was aborted". A reader that stops sending bytes, with no caller deadline, waits indefinitely. ## What Changed - `native-harness-backup-storage.ts`: add `downloadIdleTimeoutMs` to `restoreDurableNativeHarnessBackup` and `verifyDurableNativeHarnessBackup`. When set, each durable `getObject` (the backup object and the provider artifacts) runs under an idle timer. Each received chunk renews the timer. The timer is active only while a read is in flight, so local decrypt, digest, extraction and copy are never timed. On expiry the whole restore aborts and throws `native_harness_storage_download_stalled`. Caller cancellation keeps its own error. - `verifyRecoverableNativeHarnessBackup` and the executor's `restoreDurableHarnessForExecution` use the five-minute idle limit (`NATIVE_HARNESS_DOWNLOAD_IDLE_TIMEOUT_MS`) instead of `AbortSignal.timeout(300_000)`. - The whole restore or verification keeps a 30-minute outer limit. It fails with `native_harness_storage_restore_deadline_exceeded`, so a hung local step stays bounded and named. - The executor passes the startup Stop signal (`providerPackSourceAbort`) into the restore, so Stop (also a Stop that came before the restore began) ends it. Only a startup that already completed does not reuse its signal. The restore checks the signal after the local copy and before it publishes `current`. - The wrapped storage is a `Proxy` that replaces only `getObject` and forwards every other member to the real service. - `doc/DEVELOPING.md`: document the idle limit and the named code. Previous-release story (`rules/upgrade-safe-run-state.mdc`): no persisted shape changes. Backups, artifact rows, receipts and checkpoints saved by the previous release restore unchanged. A deploy that lands during a restore stops the process. The new process starts the restore again from the saved durable backup under the new limit. The existing hash, size, encryption and atomic publication checks are unchanged. ## Verification Run from `server/`: - `pnpm exec vitest run src/services/native-runtime/native-harness-backup-download-deadline.integration.test.ts` — 4 tests passed (embedded PostgreSQL, local-disk storage, an OpenCode backup with provider artifacts). The tests are in a focused suite, so the existing 990-line storage suite does not grow. - `pnpm exec vitest run src/services/native-runtime/native-harness-backup-storage.integration.test.ts` — 68 existing tests passed. - New: "keeps restoring a backup whose download outlasts the idle window while bytes keep arriving". With the former total-deadline policy, the same slow transfer fails with "The operation was aborted" and publishes nothing (the staging message). With the idle limit, the restore completes after more than twice the window, restores the session and the OpenCode launch artifacts, and verification also completes. - New: "names a download that stops sending bytes and publishes no partial backup". It checks restore and verify, checks that no `current` backup is published, and then restores the same saved backup successfully. - New: "stops at the outer bound during local copy and publishes nothing". - New: "lets Stop end a progressing download with its own reason, including a Stop before it starts". - RED: with the production files reverted to `origin/master`, the stall test does not finish and times out after 10 s, because no stall is detected. The other two tests pass on the old code, because the total deadline was the executor's caller policy. The first test reproduces that policy directly. - `pnpm exec vitest run` on `native-harness-online-preservation`, `native-harness-filesystem-independence` and `native-controller-migration` integration suites passed. `native-warm-checkpoint-ownership.integration.test.ts` fails locally on master as well, because the local Rust `paperclip-runnerd` binary is not built. - `node scripts/typecheck-layers.mjs` passed for all layers. Not verified: a live resumed run on staging with this build. The staging evidence shows where the time goes (before the coordinator claim, on resumed sessions only) but does not show which restore step was slowest. The fix removes the total deadline for every step on that path. ## Risks - A restore that progresses very slowly can now take longer than five minutes. Before, it failed. The run still stops on caller cancellation. - A local step that hangs (for example a stalled disk) is now cut off by the 30-minute outer limit instead of 5 minutes. A stalled local writer also pauses the read through backpressure, so it can be reported as `native_harness_storage_download_stalled`. - A failure before the coordinator claim still enters the existing saved-work reconciliation path. This PR does not change that classification. It removes the cause of these failures. ## Model Used - Provider: Anthropic. Model: Claude Opus 5.5 (`claude-opus-5-5`), with tool use (shell, file edits, read-only browser for API inspection) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 14:34:57 -06:00
DottaandClaude Opus 5.5	f439150cb3	fix(recovery): limit the null-host owner fallback to single-host servers and record each use (#19883 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - #19868 and #19871 made restart recovery judge a recorded pid only in the PID space that recorded it > - Rows saved before those host columns existed have no recorded host. They kept the previous local dead-pid proof, so in-flight native runs would not be stranded at upgrade > - On an ephemeral (multi-host) server that fallback is unsafe: the row may have been written on another host > - This pull request limits the fallback to single-host servers, records every use, and gives ephemeral servers a working board action > - The benefit is that single-host upgrades still recover on their own, multi-host servers never accept a cross-host proof, and the next release can remove the fallback with evidence ## Linked Issues or Issue Description Bug description After #19868 and #19871, a `native_run_finalizations` row with `controller_host_hash = null`, or a run with `process_host_hash = null`, still accepted "the pid is dead here" as proof on every server. That includes an ephemeral server, where the row may come from another host. Nothing recorded how often the fallback was used, so it had no safe removal path. Expected behavior - Single-host server: the fallback applies for one release, and each use writes a `legacy_host_unknown_dead_pid_proof` activity-log entry. - Ephemeral server (`PAPERCLIP_EPHEMERAL_SERVER=true`): the row is unprovable, and the board has an action to confirm the previous server stopped. - A mismatched host is always unprovable. Refs #19868, #19871, #19846 ## What Changed - `native-restart-recovery.ts` `evaluateNativeControllerTakeover`: - A coordinator row with `controllerHostHash: null` returns `controller_host_unrecorded` when `ephemeralServerEnabled()` is true. - Otherwise a pid proof from that row carries `legacyHostFallback: true`. - `operator_attested` is accepted as proof (`operator_attested_controller_stopped`). - Receipt-built owners, which have no host key, are unchanged. - New `recordLegacyHostFallbackProof` writes the activity-log entry. It is not a run-log event: run events are progress evidence for restart recovery's crash-loop credit, and the record must not change any recovery decision. It is called from restart recovery, the dead-controller result-lease release, native retirement, and remote warm-session recovery. - New `native-controller-host-recovery.ts`: - `requestNativeControllerStopConfirmation` (board-owned `native_controller_host_unverified` recovery action, idempotent, never displaces another active action). - `confirmNativeControllerStopped`. It verifies the exact recorded controller and stamps `operator_attested` with compare-and-set. It then resolves the action, wakes live-run recovery, and logs `issue.native_controller_stop_confirmed`. - Route `POST /issues/:id/recovery-actions/confirm-controller-stopped`. It is board-only and requires `runtime:confirm_owner_stop`. Generic `restored`/`false_positive` resolution of this cause is refused. A shared `confirmNativeControllerStoppedSchema` validates the body. - UI: `WorkspaceExportRecovery` reuses the "Confirm the previous server stopped" control for this cause, and the generic recovery card hides it. Adds the `issuesApi.confirmControllerStopped` client call. - `recovery/service.ts` orphan reaper: a null `process_host_hash` is judged only when not ephemeral. A successful use records the entry best-effort, after the terminal write. On an ephemeral server the run is left to its existing Stop control. - `ephemeralServerEnabled` moves to a leaf `ephemeral-server-mode.ts` (re-exported by `ephemeral-server-contract.ts`). Importing the contract module from recovery code created an import cycle in the runner child process. - The new route is registered in OpenAPI. - `doc/execution-semantics.md`: documents the host rule and the one-release sunset. ## Verification - `native-restart-recovery.test.ts`: null host on a single-host server is released with the flag. Null host on an ephemeral server is unprovable. A mismatched host is unprovable in both modes. An attested controller is accepted. Receipt owners are unchanged. - New `native-controller-host-recovery.test.ts` (embedded Postgres): - The fallback writes the activity-log entry and no run event. - Ephemeral mode creates exactly one board action. - A wrong run id is refused. - Confirmation stamps the attestation, after which takeover is allowed. The action resolves and the activity is logged. A second confirmation is refused. - `recovery-stale-issue-lock-sweep.test.ts`: on a single-host server a null host is terminalized and records one event. On an ephemeral server it is left alone with no event. The existing audit-failure case still clears the lock. - `WorkspaceExportRecovery.control.test.tsx`: the controller confirmation posts to `confirmControllerStopped` and never to the copyback endpoint. - Locally: these suites plus `remote-warm-session-recovery`, `native-controller-recovery.integration` and `native-startup-cleanup-recovery` (136 passed), and after the CI fixes `native-runner-restart-preparation` and `native-runner-restart-recovery` integration, `openapi-routes` and the suites above (256 passed); the UI recovery suites (52 passed); `node scripts/typecheck-layers.mjs` with no errors in the touched files; `pnpm check:token-gates` clean. ## Risks - Previous-release story: no schema change. Null-host rows saved by earlier releases recover as before on single-host servers, now with an event, and wait for the board on ephemeral servers. The next release removes the fallback; the event count shows whether any rows remain. - On an ephemeral server, a null-host orphaned run process is not reaped automatically. The operator uses Stop. That is the intended "unprovable" outcome. - A board confirmation is an attestation. It applies only to the exact recorded controller (boot id + pid) and only while the host is still unrecorded. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 13:25:47 -06:00
DottaandClaude Opus 5.5	0189fedffc	fix(native): bound an idle warm local runner whose controller died (#19879 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A warm runner (`lifecycleMode: warm`) keeps runnerd alive between runs so a follow-up turn starts fast. The follow-up default flip (next PR) makes warm the default for local Codex and OpenCode agents. > - The controller owns a warm runner's idle window. Runnerd ignores the idle timeout itself. > - A graceful controller shutdown suspends idle warm sessions. A controller that dies (SIGKILL, crash) does not. Local runners launch with no process reconnect grace, so the idle runner kept running until its 1-hour connection lease ran out, owned by nobody. > - This pull request gives warm local runners the 7-minute reconnect grace that remote runners already use. > - The benefit: idle runners are reaped by proof within a bounded time, never left orphaned, before warm becomes the default. ## Linked Issues or Issue Description Refs #19874 (warm completion contract), #19846 (TTFT baseline), #19852. What existing behavior does this improve? Ownership of an idle warm local runner after its controller dies. Current behavior `runnerProcessReconnectGraceMs` is `undefined` for local targets, so runnerd launches without `--reconnect-grace-ms`. An idle warm runner whose controller was killed stays alive until its connection lease (one hour) expires. The restart-chaos harness cannot show this: its K1 kills the controller's whole process tree, runner included. Proposed behavior `localRunnerProcessReconnectGraceMs(policy)` returns the remote grace (7 minutes) for a warm policy and `undefined` for per-turn. After the grace, runnerd records `transport_reconnect_grace_exceeded`, stops its provider, keeps durable state and exits. The next run resumes cold. Reason and benefit The restart plan requires idle runners to be reaped by proof, never orphaned. Seven minutes is far longer than a controller restart (13–20 s locally) and the long-outage fault holds (3–5 minutes), so a live turn still reattaches. ## What Changed - `native-runner-recovery-contract.ts`: `localRunnerProcessReconnectGraceMs`. - `native-session-executor.ts`: the local runnerd transport passes it as `runnerProcessReconnectGraceMs`. Remote targets keep `null` (lease-owned). - `runnerd-warm-idle-orphan.test.ts` (new): real runnerd and fake Codex. An idle warm runner, after its turn completed, whose controller detaches. - `doc/DEVELOPING.md`: the grace and its bound. ## Verification - `pnpm --filter @paperclipai/paperclip-runner exec vitest run src/live/runnerd-warm-idle-orphan.test.ts`: with a process grace, the idle runner exits with `transport_reconnect_grace_exceeded` and the next transport resumes the same provider session cold. With no grace (`null`, today's local launch), the runner is still alive after the controller left. - `cd server && npx vitest run src/services/native-runtime/native-runner-recovery-contract.test.ts`: warm gets the remote grace; per-turn keeps none; the grace outlasts a controller restart and a long outage. - `tsc --noEmit` passes for `server` and `packages/paperclip-runner`. - Local restart measurement with warm runners (K1/K2, codex, claude, opencode, n = 3 each): every interrupted run recovered, with recovery latency equal or lower than per-turn. The chaos waves with this change are in the default-flip PR. ## Risks - A warm local runner whose controller stays down for more than 7 minutes exits. A turn in progress then resumes through `resume_dead_runner`, the same path as a controller crash that kills the runner. Remote runners already behave this way. - Previous-release story (`rules/upgrade-safe-run-state.mdc`): no persisted state changes. A warm runner launched by the previous release keeps the launch arguments it started with (no grace) until it exits. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edit, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 13:11:18 -06:00
DottaandClaude Opus 5.5	68db7ed9e3	fix(recovery): never accept a dead local pid as proof for an owner on another host (#19868 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - After a restart, recovery decides that an in-progress record's owner is gone and takes it over > - Two of those proofs use `kill(pid, 0)` on the local host. A pid that is dead here says nothing about a process on another host or in another PID namespace > - So a controller or runtime-control owner that is still alive elsewhere can be taken over while it still runs > - This pull request records the owner's PID space (the existing `controller-process-identity` host hash) and judges a pid only in that same space > - The benefit is that a different host is never proof. Those records follow their existing lease or TTL instead ## Linked Issues or Issue Description Bug description The Phase 0 ownership inventory (finding 2) found three proofs that accept "the pid is dead here" as proof for an owner on another host: `evaluateNativeControllerTakeover` (`native_run_finalizations` has no host column), the runtime-control sweep (`workspace-operations.ts`), and the orphan reaper `terminalizeOrphanedRunningRun`. Only `controllerProcessExited` checked the host. Steps to reproduce 1. Run two control-plane processes on different hosts, or in different PID namespaces, against one database. 2. Let host A own a native finalization lease or a runtime-control operation. 3. On host B, run restart recovery. If A's pid number is not alive on B, B takes over the live owner. Expected behavior An owner whose PID space differs from this process's, or cannot be read, is unprovable. Its record waits for its lease, fence, or TTL. Refs #19846 ## What Changed - Migration `0441`: add nullable `native_run_finalizations.controller_host_hash`. - `controller-process-identity.ts`: export `currentProcessSpaceHash()` and `sameProcessSpace()`. An unreadable space is stored as `"unknown"`, which never matches. - `native-restart-recovery.ts`: `currentNativeControllerIdentity` carries `hostHash`, which both coordinator claim writes (`native-restart-recovery.ts`, `native-session-executor.ts`) store. `evaluateNativeControllerTakeover` returns `controller_host_unprovable` before any pid probe when the recorded space is not this one. `native-dead-controller-result-lease.ts` selects the column. - `workspace-operations.ts`: runtime-control owner stamps record `hostHash`. `ownerProcessProvenGone` requires the same PID space (EPERM = alive), otherwise the existing staleness TTL applies. - Not in this PR: the orphan reaper `terminalizeOrphanedRunningRun`. `heartbeat_runs` has no host for `process_pid`, and its writers are spread across legacy and native paths. It follows as a separate PR with its own column. ## Verification - `npx vitest run src/services/native-runtime/native-restart-recovery.test.ts src/__tests__/workspace-runtime-control-recovery.test.ts`: 55 passed. New cases cover another host and an unreadable space (dead and recycled pid) for takeover, plus another host, unreadable space, and legacy stamp for the sweep. The sweep cases also confirm that the TTL still reclaims after the heartbeat lapses. - Red before the fix: the 5 new cases fail on the old code. Green after. - `native-controller-recovery.integration.test.ts`, `native-failed-run-owner-release.test.ts`, `native-startup-cleanup-recovery.test.ts`: 71 passed. - `pnpm db:generate` produced `0441_lucky_chameleon.sql` (one `ADD COLUMN`) and pruned to five snapshots. `pnpm --filter @paperclipai/db check:migrations` passed. Server `tsc --noEmit` shows no errors in the changed files. ## Risks - Previous-release story: `native_run_finalizations` rows saved before this column have `controller_host_hash = null`. They keep the previous local dead-pid proof for this release. Making them unprovable would leave every native run in flight at the upgrade in `awaiting_evidence` with no TTL, because local takeovers have no lease fallback. Rows written by this release onward are strict. Runtime-control stamps without `hostHash` are strict now, because that path has a staleness TTL. - Owners built from other receipts (workspace finalization owner, workspace stop proof) pass no host and keep the previous behavior. Track 2's shared `OwnerStamp` should absorb them. - A new row on a host that really moved (redeploy onto a new machine) waits for its lease or an operator instead of being taken over by pid. That is the intended "different host is never proof" rule. - No new I/O before `turn.start`. The hash is computed once at module load. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 11:14:48 -06:00
DottaandClaude Opus 5.5	a9ac2e2039	feat(native): split local runner startup into stage, launch and session-load spans (#19852 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - When a person messages an agent, the agent must start its turn fast. Time to first token (TTFT) and agent boot are first-class metrics. > - The Phase 0 baseline (#19846) found that local runner startup costs 1.6–17 s at p50 and up to 50 s at p95. It is the largest part of TTFT. > - The run log records that startup as one span (`runner.session.bootstrap` or `runner.session.resume`). We cannot see if the time goes to staging, to the `paperclip-runnerd` process launch, or to the provider session load. > - This pull request splits local runner startup into three sub-spans in the run log. > - The benefit: the follow-up work to cut follow-up-turn TTFT can measure each step before and after. ## Linked Issues or Issue Description Refs #19846 (TTFT baseline, finding 1 and gap "No sub-spans exist inside local runner startup"). Refs #19838. What existing behavior does this improve? The run-log performance spans for local native runner startup. Current behavior The local path records one `runner.session.bootstrap` or `runner.session.resume` span that covers runtime staging, the runner process launch and the provider handshake together. The remote path already records `runner.runtime.stage` and `runner.process.launch`; the local path does not. Proposed behavior The local path records three sequential child spans of `runner.session.startup`: `runner.runtime.stage`, `runner.process.launch` and `provider.session.load`. Reason and benefit The TTFT work needs to know which step dominates before it changes the runner lifecycle. Without these spans, a change to warm runner reuse cannot show which cost it removes. ## What Changed - `RunnerdCodexTransport` has a new optional `onStartupStep` callback. `#start` and `#resume` time three milestones and call it synchronously. A failing callback never changes startup. A nested startup (exact adoption from `thread/start`) keeps the outer clock. - `runner.runtime.stage`: launch preparation before the runner process exists. - `runner.process.launch`: spawn until runnerd accepts the run (`run.prepare`, or the recovery attachment on resume). An adopted live runner records no launch. This keeps the existing meaning in `summarizeStartupReuse`: a launch span means a fresh runner. - `provider.session.load`: `session.open` (or the recovery probe) until the provider identity is confirmed. - `recordNativeRunnerStartupStep` (`native-run-trace.ts`) records each step as a `run.performance.span` row with `mode` and `target: "local"`. It uses `trace.defer`, so the run-log write never delays `turn.start`. - The native executor passes the callback only for a local target. Remote launchers keep their own spans, so there are no duplicates. - `redaction.ts`: allowlist `runner.process.launch` and `provider.session.load` as native span names. - `doc/run-log-events.md`: new section "Local runner startup sub-steps". Data path (AGENTS.md §5.7): run log only, plus Observability spans when an operator sets an OTLP endpoint. The attributes (`mode`, `target`, `outcome`) are already in the closed span-attribute allowlist. No Telemetry change. ## Verification - `pnpm --filter @paperclipai/paperclip-runner exec vitest run src/live/runnerd-codex-transport-startup-steps.test.ts`: a real runnerd and the fake Codex fixture. Bootstrap and a cold resume each report stage, launch and session load in order, with `ok` outcomes and no overlap. A throwing sink does not change startup. - `cd server && npx vitest run src/services/native-runtime/native-run-trace.test.ts src/__tests__/redaction.test.ts`: the rows have the correct parent, offsets, durations and failed level. A stalled run-log sink does not block the caller. The rows settle before `task.provider_session.measured`. The names survive redaction. - The existing `runnerd-codex-transport*` suites pass (412 tests). Under host load (load average 16–27), the first full run had 7 timing failures in suites that do not set the callback. Each failing file passed when I ran it again alone. `runnerd-codex-transport-settlement` passed 3/3 for the failing case alone. - `tsc --noEmit` passes for `packages/paperclip-runner` and `server`. ## Risks - Low risk. Without `onStartupStep`, the transport behaves as before: the clock is not created and each milestone call is an optional-chain no-op. - More run-log rows: three per local runner startup. - The step boundaries are controller-side observations. `runner.process.launch` includes the runnerd authentication and the `run.prepare` acknowledgement, not only the OS spawn. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edit, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 10:57:31 -06:00
DottaandClaude Opus 5.5	824b25654c	fix(ui): keep a Retry for retained-file holds whose source cannot be verified (#19840 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - When a native run finishes but its files cannot be saved, the task stays blocked behind a retained-file recovery hold. > - The server records that hold as blocked and retryable, and the task notice tells the user to "Retry to try again". > - The task page hides its own Retry for these holds and relies on the retained-file recovery panel instead. > - That panel only binds the recovery record when a read-only source inspection returns `verified`. > - Inspection can return `blocked` for normal reasons: a sandbox that predates ownership receipts (state saved by an older release), or a workspace that moved on after its files were recovered. > - Then neither surface offers an action, and "Check file recovery status" always answers "File recovery changed. Refresh the task", also after a reload. > - This pull request binds the record by its own identity, uses inspection only to gate the physical file-recovery request, and keeps the task Retry visible whenever the panel has no control of its own. ## Linked Issues or Issue Description Refs #18848 (the change that added the retained-file recovery panel and hid the generic Retry for these holds). What happened? A blocked task with an active `native_workspace_sync_out_retry_exhausted` hold showed "The saved files need recovery before work can continue", "The recorded source is unavailable or has changed", and "Saving the agent's files failed after automatic retries. Retry to try again". It showed no Retry and no recover button. "Check file recovery status" always answered "File recovery changed. Refresh the task to inspect the current recovery", and a reload did not change it. The hold was saved by an earlier release: `systemRecovery` exhausted with `allocation_ownership_ambiguous` / `service_required`, and `workspaceRecovery` blocked with `retryable: true`. The source inspection endpoint returned `status: "blocked"` for it. A second task whose files were already recovered showed the same "need recovery" and "has changed" text, because its inspection reported a reference mismatch after recovery. Expected behavior A retryable hold always shows a working Retry or recover control. A notice that says "Retry" has a Retry button. Status checks never loop on "changed, refresh". Recovered files show as recovered. When the source cannot be verified, the page names the reason and who must inspect it. Steps to reproduce 1. Use an active retained-export hold whose `workspaceRecovery` is blocked and retryable, and whose `systemRecovery` is exhausted. 2. Make the source inspection (`GET /api/issues/:id/recovery-actions?inspectSourceRunId=…&inspectSourceActionId=…`) return `status: "blocked"`. 3. Open the task as a board operator. 4. Before this change, the page shows no Retry and no recover button. "Check file recovery status" reports "File recovery changed. Refresh the task". Paperclip version or commit Observed on build `8812c19`. The regressions fail on master `0f5ce5050e` before the production edits. ## What Changed - `WorkspaceExportRecovery`: bind the recovery record by action, run and schema identity, not by inspection status. Inspection status now gates only the physical "Recover saved files" request (and the request re-checks it before POST). - `WorkspaceExportRecovery`: report through `onRecoveryControlChange` whether the panel offers its own control. It reports `true` only while the source is loading or verified and the files are not yet recovered, or while it shows the previous-server stop confirmation. - `IssueDetail`: hide the task Retry for retained-export holds only while the panel reports that it owns the control. Otherwise the existing task Retry (`POST /issues/:id/retry`) shows. The server renews a bounded recovery generation and re-verifies the source. - Show recovered files as recovered even when inspection no longer matches after recovery. - Name an unverifiable source in plain words (with the code for other reasons). State that Paperclip does not apply files it can't verify. - Recovery text never asks a person to inspect a sandbox or environment, and never names an owner or administrator as the one who must act. This covers the retained-file panel, the run source inspector, the copyback wait, and the native cleanup quarantine message. The execution blocker and the recovery action read model describe a spent recovery budget from its current state, so text saved by older releases ("If Retry keeps failing, inspect the run and its environment") is not shown. - Task Retry uses the existing `createUuid()` helper, so it works on plain-HTTP previews without `crypto.randomUUID`. An accepted Retry also refreshes the inspected file-recovery state. After a failed accepted request, the unverified-source reason stays visible; it hides only while recovery runs. - When the recovery record was replaced, "Check file recovery status" reloads the task detail instead of telling the user to refresh. - Tests: component regressions for an old-release hold, a recovered hold with a mismatched source, verified ownership, and a replaced record. A page regression proves Retry is shown and calls `retryTask` for unverified and stale sources. A server safety test proves that Retry for the old-release hold shape is accepted, opens one new bounded generation, queues file recovery again, keeps the hold active, and is idempotent. ## Verification - RED (master `0f5ce5050e` production files, new tests): 5 component tests fail. The failure output shows the exact staging text: "Check file recovery status", "File recovery changed. Refresh the task to inspect the current recovery.", "The recorded source is unavailable or has changed." 2 page tests (unverified and stale) fail with `expected [ 'Check file recovery status', … ] to include 'Retry'`. - GREEN: `cd ui && npx vitest run src/components/WorkspaceExportRecovery.control.test.tsx src/components/WorkspaceExportRecovery.test.tsx src/components/RunWorkspaceRecoverySurface.test.tsx src/components/ExecutionBlockerNotice.test.tsx src/pages/IssueDetail.test.tsx` passes 320 tests. - Server: `cd server && npx vitest run src/services/task-retry.test.ts` passes 81 tests, including the new old-release safety case. This is safety evidence, not a red: the server path was already correct, and the defect was in the UI. - Full UI suite: 11564 pass. 4 tests in `IssuesList.test.tsx` and `CompanyEnvironments.test.tsx` failed under full-suite load and pass in isolation (106/106). They are unrelated to this change. - `pnpm check:token-gates` is clean. `node scripts/run-tsc.mjs -p ui/tsconfig.json --noEmit` reports no errors in the touched files. The remaining errors are local artifacts that were not built (plugin SDK UI entry, generated Storybook catalog). - Staging was read only (GET requests). The inspect request did not loop in a fresh tab: two calls in 30 seconds. Most of the tab's request volume is periodic company issue-list refetches, which this change does not touch. ## Risks - Low risk. This is a UI-only behavior change. Server authorization, source verification, stop receipts, and the bounded recovery policy are unchanged and remain authoritative. - The task Retry is now visible for retained-export holds whose source does not verify. Retry renews one recovery generation. The worker re-runs the same source-identity checks, so Retry cannot bypass an ownership refusal. If the refusal is permanent (for example an ambiguous sandbox allocation), the hold returns to exhausted and the page names the reason. An automatic fallback for such sources is follow-up work. - Wording: a parametrized UI test renders every recovery state and asserts that no administrator, owner, or "inspect the environment" copy appears. A server test proves that a stored old-release message is replaced by state-derived text. Both failed before the change. - Previous-release story: the old-release hold shape (board-owned, `systemRecovery` revision 5 exhausted with `service_required`, `workspaceRecovery` blocked and retryable) is covered by component, page, and server tests. No persisted state shape changes. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through Claude Code with tool use (repository search, file edits, shell, test execution, and a read-only browser). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 10:24:03 -06:00
DottaandClaude Opus 5.5	630411af53	fix(recovery): require a 1 s start-time gap and treat EPERM as alive in owner proofs (#19848 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - After a restart, recovery code decides whether the process that owns an in-progress run is gone > - Two proofs compare process start times to the millisecond, but `ps -o lstart` on macOS and BSD reports whole seconds > - So a live owner can look like a reused PID, and recovery can release it while it still runs > - Signal-0 handling also disagrees: one check treats EPERM as dead, while all others treat it as alive > - This pull request puts one start-time rule and one EPERM rule in `controller-process-identity.ts` and uses them at every site the Phase 0 ownership inventory named > - The benefit is that no proof releases a live owner because of clock precision or a permission error ## Linked Issues or Issue Description Bug description The Phase 0 ownership inventory (finding 1) found inconsistent "owner gone" proofs. `native-failed-run-owner-release.ts` (`recordedProcessMayBeLive`, `runProcessMayBeLive`) and `native-session-executor.ts` (`verifiedRecoveryProcessIsAlive`) treat any start-time difference of 1 ms or more as a reused PID. `native-restart-recovery.ts` and `controllerProcessExited` require a gap of at least 1 s. `local-service-supervisor.ts` (`isPidAlive`, `isProcessGroupAlive`) treats EPERM as dead. Steps to reproduce 1. On macOS, record a live process start time with millisecond precision. 2. Run the abandoned-owner release pass, or the recovered-runner liveness check, against that live PID. 3. The pass releases the owner, or the check reports the runner as gone. Expected behavior A sub-second start-time gap is ambiguous and keeps the owner. EPERM means the process exists. Refs #19846 ## What Changed - `controller-process-identity.ts`: export `processExists` (EPERM = alive) and `processStartsDefinitelyDiffer` (gap >= 1 s; a missing time proves nothing). `controllerProcessExited` uses the new helper. - `native-failed-run-owner-release.ts`: the controller and run-process checks use the shared helpers instead of exact millisecond equality. - `native-session-executor.ts`: `verifiedRecoveryProcessIsAlive` accepts a start time within 1 s as the same runner. Its signal path uses the same check. - `native-restart-recovery.ts`: `definitivelyDifferentProcessStart` delegates to the shared helper. No behavior change. - `local-service-supervisor.ts`: `isPidAlive` and `isProcessGroupAlive` treat EPERM as alive. ## Verification - New `controller-process-identity.test.ts`: the 1 s boundary, EPERM and ESRCH for every signal-0 check, and a live spawned runner whose recorded start is 500 ms off. - `native-failed-run-owner-release.test.ts`: two new cases (controller and run process) with a live child and a recorded start 500 ms off. The owner stays `observed`. - Red before the fix: with the old call sites, all 4 new cases fail (`releaseAbandonedNativeOwners` returns 1 and releases the live owner; EPERM returns false; the live runner reads as gone). Green after the fix. - `npx vitest run src/services/controller-process-identity.test.ts src/services/native-runtime/native-failed-run-owner-release.test.ts src/services/native-runtime/native-restart-recovery.test.ts src/__tests__/local-service-supervisor.test.ts`: 59 passed. ## Risks - Low risk. The change only makes proofs stricter: a sub-second gap or EPERM now keeps an owner that the old code released. Such a row waits for the existing lease, TTL, or real process exit. - A recycled PID whose new process started within 1 s of the recorded one now stays held until it exits. That is the same rule the takeover path already used. - Previous-release story: no persisted shape changes. Rows saved by the previous release are evaluated with the stricter rule. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 10:06:35 -06:00
DottaandClaude Opus 5.5	e2d490c61b	feat(observability): anchor native time to first token at the accepted wake (#19838 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When someone messages an agent, the reply must start fast. The durable ownership plan (#19831) makes time to first token (TTFT) a hard invariant. > - Phase 0 of that plan records a TTFT baseline per adapter and environment, measured from the moment the message or wake is accepted. > - Today `task.time_to_first_token` starts at the wake comment, the answered question, or the run row's creation. A wake request that was accepted earlier (deferred, coalesced or retried) is not part of the measurement, and the span does not say which start it used. > - This pull request moves the start back to the earliest accepted input and names it on the span. > - The benefit is that the baseline and later budgets measure what the user waits for, with no new work before `turn.start`. ## Linked Issues or Issue Description Refs #19831 What is the problem? `task.time_to_first_token` does not include the time between an accepted wake request and the creation of its run row. It also does not record which input it started from. What is the expected outcome? The span starts at the earliest accepted input and carries an `anchor` attribute: `comment`, `question_response`, `wake_request` or `run_created`. Why does it matter? Phase 0 of the durable ownership plan sets TTFT budgets from this baseline. A start that skips queue time hides the latency that deferred wakes add. ## What Changed - `resolveNativeTimeToFirstTokenStart` (pure) in `native-run-trace.ts` picks the start and the anchor. `anchor` joins the closed span-attribute allowlist. - `readNativeRunWakeRequestedAtMs` reads `agent_wakeup_requests.requested_at` for the run. It never throws. - `native-session-executor.ts` calls it only after the first deliverable text, so it adds no wait before `turn.start`. The span now ends at the text commit time, not after the trace write. - Updated `doc/run-log-events.md` and `doc/observability.md`. ## Verification - `cd server && npx vitest run src/services/native-runtime/native-run-trace.test.ts src/services/native-runtime/native-run-wake-anchor.test.ts` (26 pass). - `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts -t first` (41 pass). The existing TTFT test now asserts the anchor for each ingress. ## Risks - Low risk. This changes only the run-log and OTLP span. It is not Telemetry (AGENTS.md §5.7), and it does not change runtime behavior. - One indexed primary-key read per native dispatch, after the first text. A read failure drops the anchor and keeps the old start. - TTFT values can grow for deferred wakes. That is the intended correction. `startOffsetMs` can be negative when the wake request comes before the trace start. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 09:57:30 -06:00
DottaandClaude Opus 5.5	84908d176b	fix(recovery): name the failed check in native_remote_recovery_lease_mismatch (#19839 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Remote native runs execute in a sandbox, and recovery can reattach a run to its original sandbox runner > - Reattachment checks the saved lease, workspace, and runner identity in five places. Each failed check throws the same bare `native_remote_recovery_lease_mismatch` > - On staging, 34 OpenCode runs on 31 tasks failed with this bare error in one evening. The run records could not show which check failed, so the cause could not be confirmed from the run alone > - This pull request makes every producer name the failed step and field with closed values > - The benefit is that an operator and the next investigation can see which reattachment check failed, without new logging or identifiers in the error ## Linked Issues or Issue Description Refs #19399, Refs #19733, Refs #19755, Refs #19779 What happened? Remote same-run retries failed with `native_remote_recovery_lease_mismatch` and the task was blocked for reconciliation. The error is thrown by environment acquire, environment resume, workspace preparation, workspace sync, and remote runner verification. All of them used the same bare string. The run's `error` field and its run events did not show which check failed. Steps to reproduce 1. Start reattachment for a remote native run whose saved lease, workspace reference, or runner identity no longer matches. 2. Read the run's `error`. 3. It shows only `native_remote_recovery_lease_mismatch`. Expected behavior The error names the step and the field that differed, with no identifiers, and existing consumers of the code keep working. ## What Changed - New `server/src/services/native-runtime/remote-recovery-lease-mismatch.ts`: the code constant, a closed `stage` and `field` vocabulary, and `remoteRecoveryLeaseMismatch(stage, field)`. It builds `native_remote_recovery_lease_mismatch: stage=<stage> field=<field>`. - `environment-run-orchestrator.ts` (stages `environment_acquire` and `environment_resume`), `heartbeat/workspace-preparation.ts` (`workspace_preparation`), `native-workspace-sync.ts` (`workspace_sync`), and `verifyRemoteRunnerReattachment` in `native-session-executor.ts` (`runner_verification`) now throw the detailed form. Each check keeps the same condition and order. Only the message changes. - `restart-interrupted-turn.ts` uses the shared constant. It already matches the code exactly or as the `code: ` prefix, so automatic continuation after a reattachment loss is unchanged. No other consumer matches the code. - `doc/execution-semantics.md` documents the detail format. ## Previous-release state This change adds no persisted state and changes no saved identity. Runs that a previous release failed keep the bare `native_remote_recovery_lease_mismatch` in `error`. Runs failed by this release store the detailed form. The only consumer (`isReattachmentLoss` in `restart-interrupted-turn.ts`) accepts both forms, and `workspace-recovery-retry-copyback.test.ts` covers both. ## Verification - Exact-detail assertions in the producer tests: - `server/src/__tests__/environment-run-orchestrator.test.ts`: every scope, status, expiry, provider, lease-ID, cwd, plugin-task, and resume case asserts its stage and field. Against the previous producer, 21 of these cases fail (the message was the bare code). With this change, all 57 tests pass. - `native-session-executor.test.ts`: a replaced sandbox reports `runner_verification`/`provider_lease_id`. A missing durable runner identity reports `identity_run_id` before any sandbox read. - `native-workspace-sync.test.ts`, `heartbeat/workspace-preparation.test.ts`, `native-controller-recovery.integration.test.ts` assert their exact detail. - End to end on embedded PostgreSQL: `heartbeat-native-remote-failure-retention.test.ts` records `native_remote_recovery_lease_mismatch: stage=environment_resume field=lease` on the run when the provider no longer resumes the lease, and it still requests the user's Retry once. `workspace-recovery-retry-copyback.test.ts` adds a detailed-form row that still continues once. - The affected server test files pass locally. `pnpm --filter @paperclipai/server typecheck` passes. ## Risks - Low risk. Only the error text changes. Conditions, ordering, and fail-closed behavior are unchanged. - Logs and run errors show a longer message. The detail has closed values only, so it adds no identifiers or secrets. - A first version of this PR also changed remote retry selection. Review showed that a plain resume in a new sandbox hits the remote workspace's unsaved-work hold, and that a `destroy` disposition does not prove the old sandbox is gone. That change is removed. It needs a separate design. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, run in Claude Code with tool use (shell, file editing, read-only browser access to staging). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 09:27:14 -06:00
DottaandClaude Opus 5.5	a94881d1a7	fix(heartbeat): save a comment refused by a temporary hold and route post-quarantine input to its own gate (#19829 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A user's follow-up comment or Retry must be delivered once, eventually, even when a temporary company admission hold (accounting still settling) is active > - #19741 kept claimed runs queued through such a hold and woke held work when it cleared > - A restart-chaos wave still lost a comment, because its wake was refused before admission and written as a terminal skip > - The same wave left a Retry waiting forever for a stop record of a process that never existed > - This pull request saves the refused comment as a deferred receipt, and routes input after a pre-launch quarantine failure to the quarantine's own gate > - The benefit is that the comment is delivered after the hold, and the task asks for the action that actually unblocks it instead of offering a Retry that can never start ## Linked Issues or Issue Description Refs #19741 and #19702. What happened? In a restart-chaos wave, a failed run's accounting held the company for 600 s. 1. A user commented while the task's run was live. The enqueue budget gate refused the comment's wake before it reached the deferral behind the live run. Because it was a user request with no saved receipt, it was written as `skipped` (`budget.blocked`). The live run finished without it and the task ended `done`, so the comment was lost. 2. A Retry created a run that the native cleanup-quarantine fence refused before it requested any runner launch (`native_session_cleanup_quarantined`, no process events). A second Retry then waited with `process_identity_missing` ("The previous run has no verified stop record") for a process that never existed. The task kept offering Retry, and that Retry could never start. Expected behavior A comment refused by a temporary hold stays saved and is delivered once later. Input after a run that was refused before launch is handled by the quarantine gate: Retry is not offered (send a new message), and a new message gets the verified-stop check of the previous provider session. ## What Changed - `heartbeat/queue.ts`: when the enqueue budget gate refuses a user's comment wake and the block is temporary (not a reached hard stop, per `isBoardOwnedBudgetBlock`), the wake is saved as a `deferred_issue_execution` receipt. The receipt carries the comment ids and an `executionWait` of `budget.blocked`, and the gate notifies the deferred-receipt worker and throws `PersistedExecutionWaitRefusal`. It is not written as `skipped`. Normal promotion (when the live run ends) and #19741's hold-release wakeups deliver it. - `native-local-process-stop.ts`: new `nativeRunRefusedBeforeLaunch`. It returns true for a failed native run with `native_session_cleanup_quarantined`, no recorded process, and no launch-request or identity event. - `explicit-native-continuation.ts`: such a run passes the process-identity check, so the existing quarantine branch decides. A Retry gets `cleanup_quarantined` (shown with `canRetry: false` and "send a new message"). A new message gets `verifyStoppedNativeSessionForContinuation`. Any launch request or identity record keeps the original stop proofs. - `native-session-executor.ts` (`verifyStoppedNativeSessionForContinuation`): when the source has no process record and no launch event, the verifier relies on the previous owner's stop evidence. Before this change it required the source to share the owner's process identity, so a new message stayed held even after the owner stopped. A source that carries a process identity must still match the owner exactly. ## Verification Each new test fails on `master` and passes with this change, on embedded Postgres: - `server/src/__tests__/heartbeat-held-comment-wake.test.ts`: a comment whose wake the hold refuses at enqueue is saved with a `budget.blocked` wait and is not skipped. It is delivered exactly once after accounting settles. - `server/src/services/explicit-native-continuation.test.ts`: - A run refused by the quarantine before launch routes a new message to the quarantine's stop verification and a Retry to `cleanup_quarantined`. These failed on `master` with `process_identity_missing`. - Guards (green on both): a quarantined run that requested a launch, and another failure without a process record, still require `process_identity_missing`. - `server/src/services/quarantined-task-retry.test.ts`: a never-launched quarantined source is certified once the previous owner has stopped (red on `master`). It stays held if the source requested a launch or the owner is still running. All 31 tests pass, and the `stopped native conversation physical cleanup` suite passes (46). - Related suites pass locally (6 files, 620 tests): explicit native continuation, held comment wake, queued comments routes, comment wake batching, refused assignment wake, deferred receipt work. Two `deferred-execution-receipt-work` tests failed once under parallel load and passed when run alone. ## Risks - The saved-comment path applies only to user comment wakes refused by a temporary budget hold. Hard stops, system wakes, case handoffs, durable chat requests, and existing saved receipts are unchanged. - The pre-launch rule is limited to the quarantine failure code with no launch evidence. Every other missing process identity still requires the existing stop proof. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, run as an agent in Claude Code with tool use (shell, file editing, local test execution, kept restart-chaos instance inspection). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 08:54:14 -06:00
DottaandClaude Opus 5.5	b2af45f0f1	fix(runner): let OpenAI API-key Codex agents generate images with their existing key (#19654 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - New users often connect a Codex agent with an OpenAI API key during onboarding. > - A user asked such an agent for a logo. The agent said that the built-in image generator was not available and that its fallback needed a separate API key. It then drew an SVG by hand. > - #19516 removed `--disable image_generation`. That change is correct, but it does not reach this path. The pinned Codex 0.160.0 shows its `image_gen` tool only when Codex signs in with ChatGPT. The check is `image_generation_available` in `codex-rs/core/src/tools/spec_plan.rs`. An API key never gets the tool. > - The runner keeps the provider key out of the agent's shell on purpose, so the skill's `scripts/image_gen.py` fallback cannot run. > - This pull request adds a Paperclip `generate_image` tool. The server makes the image with the agent's existing OpenAI key. The key never goes to the agent process or the shell. > - The benefit is that API-key Codex agents can make images with no second key. The spend goes to the agent's budget and the budget can stop it. ## Linked Issues or Issue Description Refs #19516 What happened? An agent connected through onboarding with an OpenAI API key (Codex harness, `gpt-5.6-sol`) could not generate images. The agent said: "The image skill's built-in generator isn't available in this run, and its fallback would require a separate API key." It then drew an SVG instead. Steps to reproduce 1. Create a company with a Codex agent that uses an OpenAI API key. 2. Ask the agent for a raster image, for example a logo PNG. 3. The agent reports that image generation is not available. It does not make the image. Expected behavior The agent makes the image with the OpenAI key that already powers it. Root cause The agent ran on the native runner's Codex app-server path, with `auth.json = {"OPENAI_API_KEY": ...}`. Codex 0.160.0 removes `image_gen.imagegen` from the model's tools unless `current_auth_uses_codex_backend()` is true. That function is true only for a ChatGPT sign-in. I proved this with the real pinned binary. I gave it the runner's isolated launch arguments and the API-key auth file, and pointed it at a local Responses endpoint. The first model request offers only `exec`, `wait`, `request_user_input` and the collaboration tools. #19516's test checked only `codex features list`, so it could not see this gate. ## What Changed - New `server/src/services/native-runtime/generate-image-tool.ts`: - The tool schema and definition. - `imageGenerationBindingForRun`. It applies only to native Codex runs on a direct OpenAI `api_key` connection with no router selection. - The OpenAI Images API call (`gpt-image-2`, `b64_json`, PNG check, 180 s timeout). The reply is read as a stream with a byte cap before it is parsed. A decoded image larger than 20 MiB is refused. - A reviewed rate card for `gpt-image-2`: text input $5, image input $8 and image output $30 per 1M tokens, from developers.openai.com/api/docs/pricing, retrieved 2026-11-06. - A create-only, symlink-confined local write. - `paperclip-runner-tool-authority.ts`: - Adds the `imageGeneration` binding. - Advertises the tool only for standard mode with a workspace. A remote target must also have an installer. Dot, Muse, Grok and native review do not get the tool. - Adds the `#generateImage` handler. Before any spend, it checks again: the run and task, budget authority, `issue:mutate`, case handoff, and the connection. - Stores a claim-before-spend per-call receipt in `resultJson.imageToolReceipts`. A retry or recovery replays the receipt and does not pay again. - Saves a `charged` receipt (usage, price, request ID and charge time) as soon as OpenAI returns. A failed save, failed accounting write or crash can then still settle the spend. A retry or recovery records the cost once and never buys again. - Records one `cost_events` row through `createCostEventInTransaction`. The row is `metered_api` with `costStatus: "estimated"`, and the budget is evaluated. The row is recorded even if saving fails. - Checks the task's project budget, as well as the company and agent budgets, before spending. - Writes content-free `runner.image_generate` activity entries: started, completed, failed and unconfirmed. - `ai-connections.ts`: adds `selectImageGeneration`. It checks again that the connection is an active, healthy OpenAI `api_key` connection with no custom routing. It also checks the responsible user's membership, the grant audience and the agent or company installation. These are the same rules that runtime selection uses. - `native-session-executor.ts`: gets the binding from the run's admitted `contextSnapshot.aiConnection`. The value comes only from the server, never from tool input. - `native-session-resume.ts`: the native tool contract changes from `v18` to `v19` and now lists `generate_image`. A persisted provider thread cannot add tools when it resumes, so older threads must rotate. - `doc/runner-api-tools.md`: a new section about image generation for OpenAI API-key Codex agents. - #19516: kept, not reverted. Its changes have no effect on the API-key path. They are still correct for Codex agents that sign in with ChatGPT, because `image_gen` works there and saves to `CODEX_HOME/generated_images`. Those agents need the read grant that #19516 added. On the API-key path, the new tool replaces the built-in tool. The tool's description tells the agent to use it instead of `image_gen` or `scripts/image_gen.py`. ## Verification Red on `master` (`e482f0f632`). Behavioural red with the first version of the new test: ``` FAIL src/services/native-runtime/generate-image-tool.test.ts > generate_image for OpenAI API-key Codex agents (N-12) > lets an API-key Codex agent make an image with its existing OpenAI key, saved in its workspace and billed AssertionError: expected [ 'connections_search', …(51) ] to include 'generate_image' ❯ src/services/native-runtime/generate-image-tool.test.ts:78:60 Tests 1 failed \| 3 passed (4) ``` The final test file on master's source also fails. The reason is that the module does not exist: `Cannot find module './generate-image-tool.js'`. Real pinned Codex binary. The new test is `packages/paperclip-runner/src/drivers/codex/codex-api-key-image-tools.test.ts`. It uses the runner's isolated launch, the API-key `auth.json` and a local Responses endpoint. It shows that the model gets no `image_gen`, `imagegen` or `image_generation` tool, and that a server-advertised `generate_image` tool does reach the model: ``` Test Files 1 passed (1) Tests 1 passed (1) ``` Green after the fix: `server/src/services/native-runtime/generate-image-tool.test.ts` uses embedded PostgreSQL and a stubbed OpenAI fetch. ``` Test Files 1 passed (1) Tests 11 passed (11) ``` The 11 tests cover these behaviours: - Main path. The tool is advertised. The request goes to `api.openai.com/v1/images/generations` with the existing key. The PNG is written to the workspace. The key does not appear in the result or in activity. One `cost_events` row is written: 50 text tokens and 1000 output tokens, which is $0.03025, estimated, from the rate card. A retry and `recover()` replay the receipt with no second fetch and no second cost row. - No binding: the tool is not offered and `execute` refuses. - Budget hard-stop. The stop pauses the agent, so the call is refused before any fetch and no file is written. - Bad grant: the call is refused before any fetch. - Project budget stop. The task's project is paused by its budget, so the call is refused before any fetch. - Charge settlement. A crash left a `charged` receipt. The retry and `recover()` record the cost once, report the image path and do not buy another image. - Oversized reply. The read stops before the body is parsed. - Save failure. The spend is still recorded. A same-call retry does not pay again and reports the failure. - Binding gate. Only Codex on a direct OpenAI API key gets the binding. A subscription, ACPX, OpenRouter or a routed connection does not. - Rate card math. - Provider errors. The error text does not contain the key, and non-PNG data is rejected. Neighbouring suites: - `src/drivers/codex` in the runner package: 24 files, 350 tests passed. - Server: `paperclip-runner-tool-authority.test.ts`, `native-session-resume.test.ts`, `fast-responses.test.ts`, `ai-connection-session.test.ts` and the new suite. 157 of 158 passed. - The one failure is `native-session-resume.test.ts › archives a damaged prior epoch … with real runnerd` (ENOENT on `authority-epochs`). It fails the same way on unmodified `master` in this local environment, so this change did not cause it. - `tsc --noEmit -p server/tsconfig.json`: clean. - `pnpm verify` exits 2 (broad). Required CI supplies the broad coverage. ## Risks - New security surface: this needs maintainer sign-off. The server now sends the agent's OpenAI API key to `api.openai.com` on the agent's request. The tool is limited in these ways: - The key stays in the server. - Each call checks run, task, budget, mutation and connection authority again. - The destination name comes from the server and must match `^[a-z0-9-]+-[0-9a-f]{8}\.png$`. The write is create-only and confined to the workspace root. On macOS it uses `O_NOFOLLOW_ANY`. On Linux it uses a `/proc` descriptor walk. - Spend. Each call costs money. The tool description tells the agent to make only what the task needs. The cost is an estimate from a rate card (`costStatus: "estimated"`), not an invoice. The rate card applies the image-output rate to all output tokens, so the estimate errs on the high side. - Checkpoint rotation. The tool-contract bump from v18 to v19 makes existing native provider threads start fresh once at their next turn. This is what the contract requires when the tool surface changes. - Pricing drift. If OpenAI changes `gpt-image-2` prices, the rate card must be updated. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, sub-agents) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 07:48:39 -06:00
DottaandPaperclip	909f4d1a33	fix(native): recover exact historical provider pack code (#19779 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native sessions retain the provider pack that admitted their saved state. > - An image update can replace that pack's executable, proxy, or contract. > - Older sessions can predate the durable three-file launch artifact receipt. > - The current image's archive cannot satisfy a different pinned manifest. > - This pull request obtains the exact public code from its recorded release image. > - Full source, archive, and installed-file checks still control provider admission. ## Linked Issues or Issue Description Refs #19558, #19622, #19755. A saved OpenCode session can fail with `runner_remote_provider_archive_invalid` after an image update. Its original provider state and checkpoint remain valid, but its pinned pack differs from the controller's pack. The old manifest can name a release that still has the exact code in GHCR. Matching the OpenCode version does not prove matching proxy or contract bytes. ## What Changed - Resolve a pinned historical pack from the recorded source-SHA image in the public Paperclip GHCR repository. Verify the platform manifest, raw config, source identity, dedicated code layer, selected pack manifest, archive contents, paths, and hashes. - Keep a private temporary cache of verified public code. Rebuild it on an empty host. Bound each source to 4 GiB and three minutes, with two active imports or cache verifications, 32 distinct queued sources, and disk-headroom checks. Share warm-cache verification by manifest while each caller keeps its own cancellation. - Select this verified archive through the existing uploader. Preserve installed-pack and preinstalled-pack checks. Refuse unavailable historical transport before clearing the target. Keep the current-source identity guard from #19755. Refuse an oversized layer argument before any target effect. - Attach the source wait to the actual per-turn startup. Stop, lease loss, and startup completion cancel it. Keep existing Runner admission and turn limits. - Document the registry retention requirement and the distinction between public code recovery and encrypted provider-state backup. ## Verification - Genuine baseline RED: the real uploader rejects the selected old pack before any remote effect. Treatment installs the exact old code through that uploader. - Current head: all 19 source/integrity/cache/deadline controls, all 45 archive controls, and 29 actual-caller and manifest/startup controls pass (93 total). Four review regressions fail against the prior head and pass with the fixes. The server and colocated test TypeScript graph passes. Module-boundary and whitespace checks pass. - Actual public GHCR cold reconstruction: 1,054,587,008 compressed code bytes, 115.4 s, about 248 MiB maximum owner RSS. All 114 file layers and the overlay passed source validation. - Linux Node 24.22.2 fixture: the real historical pack passes the normal uploader, full installed-file verifier, and pinned OpenCode version check. It used one transfer, no application startup, no network, and no provider calls. Revision12b0d2ba passed in 220.2 s with about 282 MiB owner RSS. The earlier prototype passed in 108.5 s; both measurements are retained. The three-minute source limit and the fixture's four-minute limit were unchanged. The later review fixes add shared cache verification, hash cancellation, and an early size guard. They do not change extraction or the accepted Linux fixture input. - Earlier fixture failures are retained: unsafe hardlink classification was corrected with confined-link controls; a cached image lacked the required zstd CLI; Docker Desktop bind storage caused GNU tar directory-identity errors. The passing fixture uses the official Debian zstd binary and an owned Linux tmpfs target. Production extraction code is unchanged. - Fresh Greptile review is 5/5 on `7b1a249478` with no open findings. All required CI gates passed on the reviewed head. [CI run](https://github.com/paperclipai/paperclip/actions/runs/42353011485), [fresh review](https://github.com/paperclipai/paperclip/pull/19779#issuecomment-6704065994). No hosted session was retried by this change. ## Risks - Cold historical recovery reads about 1 GiB of public code for the measured release. Registry latency or missing images can block it. The source deadline remains three minutes. - GHCR administrators can delete release images. Source-SHA tags and this temporary cache are not a permanent code-retention guarantee. A provider-state S3 backup does not replace legacy executable bytes. - Targets must support the existing qualified archive transport and selected platform. Failed checks preserve the saved state and prevent provider dispatch. - The Linux fixture proves code installation and verification. Hosted conversation continuation still needs acceptance after normal delivery. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, repository tools, and local code execution. The host does not expose the exact deployment ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes:` / `Closes:` / `Refs:` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub references) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-07 02:06:02 -06:00
Nicky LeachandPaperclip	beb3e55072	Continue a released Runner after confirmed graceful eviction (#19760 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Core can release a completed Runner while its host retains the session. > - A host can later retire that Runner under its retention policy. > - Graceful retirement saves a suspended session before physical shutdown. > - A cached Core owner still expects the old physical task and rejects a new allocation. > - This pull request retires that dead local controller after verified stop. > - The benefit is ordinary session continuation for Codex and qualified Claude. ## Linked Issues or Issue Description Refs #19611, #19683, #19196, #19733 and #19157. After a host gracefully evicts a released task, the next run can receive a new allocation. A cached controller rejects it with `native_session_released_owner_scope_mismatch`. The detached-controller path can already recover the suspended session. ## What Changed - Require the exact source, old task and current account/workspace scope before releasing a cached controller. - Use the existing physical-stop receipt checks. Detach only the local controller, then use the same suspended-state admission as the detached path. - Preserve the current run's instruction reservation. Retain a failed detach for cleanup. - Add actual Core, PostgreSQL and native Runner tests for Codex and qualified Claude through ACPX. Route them once through the native CI lane. ## Verification - The original real-entry Codex case failed at the cached-task scope check after native retirement, confirmed process exit and a recorded stop receipt. - The current head's native CI lane passed all 16 eviction cases for Codex and qualified Claude. These check cached and detached owners, current instruction preparation, reused-allocation inspection, missing stop/state, foreign identity and non-suspended state. They verify the same provider session, two ordinary turns, one accepted source result and no replacement submit on refusal. - Local checks passed the composition matrix, 12 targeted executor controls, 43 CI lane-selection checks and server TypeScript. All eight refusal cases now assert the expected error and whether the checkpoint request or host read was reached. This addresses the first standard review's test finding. - Commands: `pnpm exec vitest run src/services/native-runtime/plugin-runner-eviction.integration.test.ts`, focused `native-session-executor.test.ts` controls, `node ../node_modules/typescript/bin/tsc --noEmit -p tsconfig.json`, and `node --test scripts/__tests__/run-vitest-stable-shard.test.mjs`. - [Full CI](https://github.com/paperclipai/paperclip/actions/runs/42349579107) passed on head `9a25709ef43365b1f1849413b935785838eca19b`. [Standard review](https://github.com/paperclipai/paperclip/pull/19760#issuecomment-6703629255) is 5/5 on this head, with no unresolved findings. Whitespace and added-line secret/PII checks passed. ## Risks - Controlled provider processes and an authenticated local host fixture model the Launcher edge. This does not qualify live vendor transcript storage, mounts or a deployment. - This covers graceful eviction before allocation or before stopped-task inspection during `describe`. Loss after that inspection or an ambiguous claim remains held for reconciliation. No unavailable reply proves physical stop. - Missing or invalid suspended state still refuses continuation. The deterministic refusal rule from #19733 remains unchanged. Abrupt death without a suspended checkpoint stays with the separate #19157 work. - No new capability, mode, schema, dependency, deployment or activation is added. The existing draft #19157 and dependent branches remain unchanged. ## Model Used OpenAI GPT-6.1 Sol (`gpt-6.1-sol`), high reasoning, through Codex with repository tools and code execution. The runtime does not expose a context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-06 23:21:45 -08:00
DottaandClaude Opus 5.5	3895195f4b	fix(native): report a retained provider pack the controller does not ship as incompatible (#19755 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Remote OpenCode runs stage a controller-owned provider pack into the sandbox before the provider starts > - Since #19558, an ordinary OpenCode continuation retains the pack that an earlier release admitted for its saved provider session > - After a controller release changes the pack, the controller no longer ships those bytes, and the continuation must stay blocked > - The configured-pack upload path did not detect that case: it read the build-time archive receipt for the old digest and failed with `runner_remote_provider_archive_invalid`, a generic, retryable error that suggests corrupt image files > - This pull request compares the configured pack's identity with the selected identity before any upload and fails as `runner_remote_provider_artifact_incompatible` > - It also puts a fixed reason in each real archive integrity error > - The benefit is a correct, permanent classification that matches the bundled-image path, no wasted retries, and a diagnosable message for real archive corruption ## Linked Issues or Issue Description Refs #19558, #19743 What happened? On a hosted deployment, two remote OpenCode runs of a review agent failed before the provider process started. The error was `runner_remote_provider_archive_invalid` (classified as a generic `native_session_interrupted`). Both runs continued an issue whose provider session was admitted by an earlier release. In both runs, the run log shows `provider_pack.compatibility`, `provider_pack.verify`, and `provider_pack.verify_preinstalled` fail, then `provider_pack.archive.verify` fails after 4–6 ms. Fresh runs of the same agent on the same controller passed pack preparation with the current pack. The cause is not a cache race. The archives are created at image build time (`Dockerfile` `runner-provider-pack` stage) and the server never rewrites them. The retained authority selects the old pack digest. `readProviderPackLayers` then compares the receipt's `providerPackDigest` (current release) with that old digest and throws `runner_remote_provider_archive_invalid`. Directory staging (no archive) instead clears the staged remote pack, uploads the different pack, and fails only at the final verification. Expected behavior A continuation pinned to a pack that the controller does not ship fails as `runner_remote_provider_artifact_incompatible`, as the bundled-image path already does ("the image provider pack did not match the bundled release identity"). It does not clear or upload anything. Real archive corruption keeps `runner_remote_provider_archive_invalid` with a fixed reason that the run report shows. Steps to reproduce 1. Build a provider pack and its archives for digest A. 2. Call `uploadVerifiedProviderPack` with that pack and a selected (retained) digest B. 3. Before this change: the call fails with `runner_remote_provider_archive_invalid` (v1 and v2 archives), or resolves after it clears and re-uploads the remote pack (directory source). ## What Changed - `uploadVerifiedProviderPack` takes the configured pack's `sourceDigest`. If it is not the selected `expectedDigest`, the call fails with `runner_remote_provider_artifact_incompatible: selected provider pack <12 hex> is not the controller's pack <12 hex>; ...`. The check runs before the remote clear, the archive read, the directory hash and any upload. - The executor passes the configured pack's identity (`readRemoteProviderPackIdentity(root, false)`, the same reader that selects the current pack). - Archive integrity errors keep the `runner_remote_provider_archive_invalid` prefix and add a fixed reason: `receipt_unreadable`, `receipt_identity_mismatch`, `receipt_invalid`, `layer_invalid`, `layers_unreadable`, `archive_unreadable`, `archive_not_file`, `size_mismatch` or `sha256_mismatch`. The run report keeps only the outer message, so the reason must be in the message. Filesystem causes stay on `cause` and do not put private paths in the message. - The v1 reader now uses the shared `verifyArchive` helper. - `doc/execution-semantics.md` documents the rule. ## Verification - Red before the fix (production files unchanged, new tests only): `npx vitest run src/services/native-runtime/provider-pack-archive.test.ts` gave 5 failed / 38 passed. The v1 and v2 cases failed with exactly `runner_remote_provider_archive_invalid`, which is the hosted error. The directory case resolved after it cleared and uploaded the remote pack. The reason tests got a bare `runner_remote_provider_archive_invalid`. - Green after the fix: the same file gave 43 passed. - `npx vitest run src/services/native-runtime/native-session-executor.test.ts`: 1245 passed. This includes the #19558 continuation cases. - `remote-provider-pack`, `remote-provider-pack-authority` and `remote-provider-pack-verification` tests: 69 passed. - `pnpm --filter @paperclipai/server typecheck`: all layers passed. - Review follow-up (`c8156841a9`): an unreadable numbered layer now reports `archive_unreadable`. The layered-corruption test deletes `0.tar.zst` and checks for that reason before any transfer. The archive file gave 43 passed. - Hosted evidence came from read-only run, event and issue-run reads only. No hosted retries, recovery or deploys. ## Risks - This change does not unblock an affected session. It makes the failure honest and permanent: `runner_remote_provider_artifact_incompatible` is already a permanent, pre-session failure code, so the run is not retried with the same result. To recover, the session needs its original artifacts (the captured provider-artifact path, which #19743 repairs for new admissions) or an authorized continuity break. - Low risk for other paths. Fresh launches select the configured pack's own identity, so the digests are equal. Restored provider artifacts and compatible installed packs return before this upload path. - Callers that match `runner_remote_provider_archive_invalid` with a substring still match. No code compares the exact message. - Not tested end to end against a hosted sandbox. The executor-level wiring of `sourceDigest` has typecheck coverage but no dedicated executor test. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic. Agentic tool use with shell, code editing, test execution and read-only authenticated browser diagnostics. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 01:10:29 -06:00
Nicky Leach	167e39c4ef	fix(runner): keep warm run.attach events on ordinary accounting (#19707 )	2026-11-06 23:09:57 -08:00
Devin Foley Claude Opus 5.5 Paperclip Paperclip	c3180398fa	fix(native): resume a remote Runner after a settled suspension fix(native): resume a remote Runner after a settled suspension A scheduled checkpoint retry can retain a remote reattach claim after closing its failed predecessor cleanly suspends and exits the Runner. Resume only that exact settled source after confirmed exit and a fresh identity/state check. Preserve pending work and uncertain-exit fences. Admit the last already-charged provider attempt and share one bounded recovery deadline with the final state read, retaining closed diagnostic labels on read failures. Cover fresh ACPX Runner relaunch on attempts two and three through actual processes and the durable close barrier. Six affected suites pass 1,579 tests, with 14 environment-gated skips; all server typecheck layers pass on Node 25.6.1. The complete provider turn continuation failure remains preserved as follow-up #19751. Refs: #19366 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-Authored-By: Paperclip <noreply@paperclip.ing> Co-Authored-By: Paperclip <Paperclip-Paperclip@users.noreply.github.com>	2026-11-06 22:39:40 -08:00
DottaandPaperclip	18b6b71d83	fix(native): refresh per-turn projections from the selected checkpoint (#19744 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Per-turn native sessions save controller checkpoints and suspended Runner backups under separate owners. > - A local controller projection can still name the first turn after the next turn has saved its checkpoint and backup. > - The old per-turn restore path selected that stale local reference instead of the newer canonical source. > - Admission then saw different turn identities and archived the valid backup with the old projection. > - This change uses the canonical selected checkpoint and the existing finished-owner proof before it replaces a stale projection. > - The benefit is that a third turn keeps the valid saved conversation and backup. ## Linked Issues or Issue Description What happened? After two successful per-turn remote runs, a third turn could fail before provider execution with unavailable provider resume state. The selected database checkpoint and suspended backup belonged to the second run. The local controller projection still belonged to the first run. Expected behavior The selected checkpoint must govern controller rehydration. Preserve the older projection and the selected run's backup. Do not overwrite current, newer, foreign, changed or unsettled ownership evidence. Steps to reproduce 1. Save two real controller checkpoints for one per-turn conversation and finish both owners. 2. Leave the first checkpoint's readable projection or external marker beside the second run's suspended backup. 3. Admit the third run through the real canonical selector and backend owner. 4. Before this change, admission archives the active backup with the stale projection and leaves no active controller state. Related: #18018 added per-turn controller restoration. #18718 refreshed warm projections. #18846 added finished-owner retirement. This change keeps the selected per-turn source from entering that retirement path because of an older projection. The open warm-only change #18436 concerns a different restart branch. ## What Changed - Resolve the canonical source before refreshing a prior per-turn controller projection. - Re-read it under the existing sorted coordinator locks. Replacing an older owner reuses finished-owner checks for both runs and verifies their full session scope. A same-source marker only rehydrates its exact selected identity before the normal retained-sandbox verification. - Recheck the exact initial local bytes after asynchronous reads and lock waits. Preserve the old control-plane directory in the existing archive, while keeping the newer backup and provider files active. - Restore a current-run external marker only from its own checkpoint before the existing bootstrap retry proof. A missing current checkpoint cannot select a previous turn. Retain readable current-run, legacy file-backed, backup integrity and provider resume checks. - Stage the existing OpenCode proxy bytes into the controlled fake-sandbox filesystem in the real ingress-retry fixture. Keep remote artifact path and digest checks unchanged. - Add real PostgreSQL/filesystem coverage for Codex and ACPX, readable projections and external markers, missing or corrupt parts, held owners, foreign scope, newer history and a current owner arriving during a real coordinator-lock wait. ## Verification - Baseline RED: the two new readable-projection cases failed against unchanged production code because active control-plane state was absent after admission. The real selector selected the second run. Setup failures were separate and are not counted as RED. - Exact final existing per-turn restore, active-owner refresh and retained-owner wait controls: 19/19 passed in 5.91s. - Exact final local source (`fd8d9d9021`, based on `131b566ca4`): 58/58 real PostgreSQL/filesystem and native Runner tests passed in 31.58s. This includes both existing ingress bootstrap retries through the real Runner and fake Codex/OpenCode providers, plus 56 focused owner cases. Cargo validated and rebuilt current source from a private APFS clone of an unqualified local cache; the original cache was untouched. A private local configuration resolves workspace DB, Shared and Runner sources without replacing behavior. The local CLI allowed a 90s setup hook; no default timeout changed. - Exact final full server and test-tree TypeScript check with Runner source paths: passed. The stock fresh-worktree check first failed on unbuilt Runner declaration entries; no product types were changed to hide that setup gap. - Exact final module-boundary and diff-whitespace checks: passed. - Native-server CI RED on the previous head: both real ingress retry cases failed because a current-run marker was returned without hydration. The aggregate gate failed from that one underlying lane. The correction preserves the own-checkpoint bootstrap path. The first local OpenCode attempt hit a controller-host fixture path guard before its intended failure; staging the same proxy bytes into the fake sandbox made both native cases pass. - Review regression RED: the real retained-sandbox fixture failed at `sandbox_resumed` on the first PR head. GREEN: the corrected owner hydrates the same-source marker and reaches the existing exact retained-lease/suspended-runner read. A mismatched item identity still refuses before any remote read. - Local environment: macOS arm64, Node 26.4.0, PostgreSQL 18.1. Provider execution is deliberately outside this admission fixture. The real three-turn workflow still needs verification on the normally deployed fixed source. ## Risks - Refresh requires settled, committed, released and stopped or fenced owners. Missing or changed evidence remains blocked. - The refresh holds existing coordinator locks while it validates bounded controller state. It performs no provider calls. - The old control-plane bytes remain in the archive. A restore write failure still fails admission; it does not reset provider state. - Final-head full CI is green, including typecheck, build, Rust, native-server, server and browser suites; Greptile is 5/5 with no unresolved threads. Hosted deployment and continuity verification remain separate acceptance steps. A successful replacement conversation is not proof of provider continuity; hosted continuity acceptance is still pending. ## Model Used OpenAI Codex, GPT-6 family, with code execution and repository tools. The host does not expose the exact backend model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-07 00:14:17 -06:00
DottaandClaude Opus 5.5	24cbea136f	fix(native): export OpenCode launch artifacts through a confined workspace copy (#19743 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native recovery saves the qualified OpenCode launch files so a cold allocation can resume the same session (#19622) > - Core reads those files from the sandbox with the provider's bulk `syncOut` > - The launch root is under the sandbox account home, not under the workspace remote dir > - Daytona, Kubernetes, and CreateOS confine every export source to the workspace remote dir, so the export always fails > - Every remote OpenCode run on a durable-storage instance now fails 40-90 s after start with `native_session_interrupted` > - This pull request exports a confined scratch copy of the three launch files from inside the workspace runtime dir > - The benefit is that remote OpenCode runs work again, and the provider confinement guard stays unchanged ## Linked Issues or Issue Description Refs #19622 What happened? After #19622, every remote OpenCode run on Daytona fails at provider admission. The run error is `Daytona sync source path escapes the workspace remote dir: /home/daytona/.paperclip-runtime/paperclip-runner/provider-artifacts/<digest>`. The error code is `native_session_interrupted`. The last successful run of the affected agent was immediately before #19622 reached the deployment. Expected behavior Core saves the qualified launch files and the run continues. The provider confinement guard still refuses any export source outside the workspace remote dir. Steps to reproduce 1. Configure an OpenCode agent on a Daytona environment with durable harness storage. 2. Start a run. 3. The run fails when Core captures the provider artifacts, before the first turn. ## What Changed - `captureRemoteNativeProviderArtifacts` now takes the workspace `remoteCwd`. - Before the export, a sandbox command copies the three launch files into a new scratch directory at `<remoteCwd>/.paperclip-runtime/paperclip-runner/provider-artifact-export/<uuid>/`. The command refuses a symlink or a non-regular file. It uses `cp -P`, so it never follows a link. - `syncOut` reads from that scratch copy. A `finally` step removes the scratch copy after success or failure. A failed removal is logged and does not replace the transfer result. - The launch root does not move. OpenCode still binds to the same absolute paths. Core still verifies each downloaded byte against the manifest SHA-256 values. - `.paperclip-runtime` is already excluded from workspace sync, so the scratch copy never enters the user workspace. - The Daytona `assertConfinedSandboxPath` guard is not changed. ## Verification - New regression `server/src/services/native-runtime/native-provider-artifact-capture-confinement.test.ts`. It runs the real sandbox commands on a local directory tree that has a home-rooted launch root and a separate workspace. Its `syncOut` applies the same workspace confinement rule as the Daytona provider. - Before the fix: it fails with `Daytona sync source path escapes the workspace remote dir: .../home/daytona/.paperclip-runtime/paperclip-runner/provider-artifacts/dfe9e4b1d660/bin/nodeCommand`. This is the staging error. - After the fix: it passes. The published bytes match, every export source is under the workspace runtime dir, and the scratch directory is empty after the capture. - Safety cases: a failed transfer removes the scratch copy and does not publish. A launch file replaced by a symlink is refused before any export, and nothing is published. - `native-session-executor.test.ts`: the durable OpenCode admission case now asserts that the capture receives `remoteCwd` and that every `syncOut` source is under `/workspace/`. - `pnpm vitest run` for the capture, confinement, remote OpenCode artifact, and session root entries tests: 14 passed. - `vitest run native-session-executor.test.ts -t "opencode\|provider artifact\|durable"`: 62 passed. - `pnpm --filter @paperclipai/server typecheck`: passed. - Not run: a live Daytona run with this revision. The fix is not deployed to staging. ## Risks - Low risk. The capture now runs two short `sh` commands in the sandbox and holds a second copy of the launch files during the transfer. The files are bounded by the existing 512 MiB limit and are deleted afterwards. - The scratch copy uses `cp -P`, `mkdir`, and `rm -rf` from POSIX `sh`. The Daytona image already uses these tools for the workspace sync. - If the sandbox stops between the copy and the removal, a scratch directory can remain under `.paperclip-runtime`. Workspace sync excludes that directory. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edit, and read-only browser access to staging). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 00:04:21 -06:00
DottaandClaude Opus 5.5	9f0ebf6f70	fix(accounting): close a run whose turn was lost on restore, and stop succeeded runs holding all work (#19724 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Budgets hold new work while the cost of completed runs is unknown > - In a restart chaos wave, a native ACPX Claude run crossed a server kill, resumed, and succeeded. Its usage receipt never became complete. > - For the 10-minute closure bound, admission refused all work in the company. This included tasks that never crossed a fault. > - This pull request fixes the receipt for a turn that was lost on restore. It also stops a succeeded run from holding all new work. > - The benefit: one late usage report no longer freezes a company, and hard stops still apply. ## Linked Issues or Issue Description Refs #19700 (the accounting closure bound that this follows). What happened? A native ACPX Claude run crossed a server SIGKILL in the middle of a turn. The Runner restored the session, and the run succeeded 54 seconds after it started. All 17 usage receipts stayed incomplete. Accounting settled the run only at the 10-minute closure bound. During those 10 minutes, admission refused all company work with "Company cannot start work while completed runs await accounting". Root cause ACPX reports provisional usage (`runDeltaComplete: false`) until a turn ends. The kill lost the turn that was active, so that turn never got a complete report. The restored Runner failed that turn with `provider_turn_lost_on_restore`. Native turn accounting (`createNativeTurnAccounting`) required every terminal turn to have usage, so this turn kept every later receipt incomplete. This was true even after the continuation turn completed with full usage. The admission gate then counted the succeeded run as a pending terminal run for the whole company. Expected behavior The receipt closes when the run's turns end. A turn that was lost on restore makes the usage a lower bound. A succeeded run with only usage outstanding does not hold other work. ## What Changed - `native-usage-accounting.ts`: a turn that failed with `provider_turn_lost_on_restore` and has no complete usage no longer holds the receipt open. The snapshot marks `accountingUsageLost`. - `native-session-executor.ts` and `native-usage-delivery.ts`: such a receipt carries the provenance note "a provider turn was lost on restore and never reported its usage; observed usage is a lower bound". - `anthropic-pricing.ts` and `codex-pricing.ts`: the rate-card pricers keep a producer's provenance note. - `native-usage-delivery.ts` and `usage-receipts.ts`: remote producer delivery keeps a `lostOnRestore` marker on the receipt's producer event, so replay also excuses the lost turn. Receipts saved without the marker still replay. - `budget-exposure.ts` and `budgets.ts`: admission waits only for pending runs that ended without succeeding (`waitingRunCount`). A succeeded run whose provider reported observed usage is bounded like a running run: its observed cost counts toward spend and reservation capacity, and its settled charge can still reach the hard stop. - `budget-exposure.ts`: a closure that timed out because of a control-plane restart keeps the unpriced-usage exemptions, whatever its token count. Restart evidence is a replaced receipt source, a native recovery transition that is not an ordinary same-run retry, or a turn lost on restore. A second chaos run showed the need. It recovered and succeeded, but its final usage report was lost with the restart, so it settled as unpriced 0 cents. The blocking policy then opened a hard incident and paused the company. A timed-out closure with no restart evidence still counts. - `run-cost-accounting.ts`: a lost receipt that carries a positive provider-reported cost settles at that cost as an estimate marked as a lower bound, not as unpriced. - `paperclip-runner-tool-authority.ts` and `budgets.ts`: a running run's own tool mutations are evaluated as admitted work. Hard stops and pauses still refuse them. Accounting holds, recovering runs and budget inputs that await evaluation do not. In a third chaos run, a live run's write to its task document was refused during a temporary hold, so the run proposed a blocked result. - `doc/SPEC-implementation.md`: documents these rules. Policy choice: a succeeded run's provider work is over, so it has less unknown spend than a running run. Running runs never held admission. Failed, cancelled, timed-out and interrupted runs still hold new work, because their provider may not have stopped. A per-agent gate would still freeze unrelated work of the same agent. A per-reservation gate does not help when `reservationCents` is 0, which was the case in this wave. ## Verification - Live: a restart chaos wave ran master plus this PR (c2a48fb1e0) with no masking. It passed 6/6, and every task crossed a restart. There were 0 accounting holds, 0 refused agent writes, 0 company pauses, and every cost settled as estimated. - Red then green: `native-usage-accounting.test.ts`, "closes a run whose earlier turn was lost on restore, as a lower bound". It uses the event sequence of the chaos run. - Red then green: `closed-native-receipt-accounting.test.ts`, "does not hold other work for a succeeded run with only usage outstanding". It also checks that the hard stop still applies. - I changed "neither pauses nor holds admission while a succeeded run's charge commits" to the new policy. A failed run in the same state still holds new work. - Red then green: `closed-native-receipt-accounting.test.ts`, "never hard-stops a blocking policy for a restart-derived timed-out receipt". It uses the zero-token row shape of the chaos run and a case with unpriced tokens. Without the fix, the company was paused. - Red then green: `runner-api.integration.test.ts`, "lets a running run write during a temporary accounting hold, but not after a hard stop". - Red then green: `closed-native-receipt-accounting.test.ts`, "does not hold other work for the s7504 succeeded row shape". - Red then green: `closed-native-receipt-accounting.test.ts`, "settles a timed-out restart receipt at its provider-reported cost, without holding or pausing". It uses the OpenCode/OpenRouter row shape from a fourth chaos run. - `anthropic-pricing.test.ts`: the rate-card provenance keeps the lower-bound note. - Local: `closed-native-receipt-accounting`, the budget suites, `native-usage-accounting`, `native-usage-delivery.integration`, `native-runner-restart-recovery.integration` and `anthropic-pricing` all pass (187 tests). Server `tsc --noEmit` is clean. `pnpm verify` asks for CI coverage because the spec changed. ## Risks - A succeeded run whose final charge is higher than its observed cost can let one more task start before the charge commits. The hard stop then pauses the scope as it does for a running run that overruns. - A run that includes a lost turn settles with a lower-bound usage figure. The provenance says so, and a later receipt still reconciles through the late-receipt path. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 23:57:48 -06:00
DottaandClaude Opus 5.5	fc7e3a6503	fix(recovery): never replace a native session after a deterministic refusal (#19733 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native agent runs keep a saved provider session, and same-run recovery resumes it after a failure > - When resume fails and the checkpoint allows it, recovery archives the saved continuity state and opens a fresh replacement session > - Recovery used that path for every resume failure, including deterministic refusals that left the saved state unchanged and repeat on every retry > - In five staging tasks a refusal (a protected launch argument, or a predecessor tool call with no result) escalated into a replacement. The remote archive move failed part way, the next attempt found no runner state, and each task was blocked for workspace repair > - This pull request stops before replacement after a closed-class refusal, keeps the saved state, and hands the run to an operator with a named reason > - The benefit is that a deterministic refusal can no longer damage saved state or open a replacement over an unsettled external action ## Linked Issues or Issue Description Refs #19659, Refs #19516, Refs #19334 What happened? Five Codex tasks on Daytona sandboxes failed on staging. The first attempt failed with a deterministic refusal: - `run.attach cannot change protected launch arguments` (four tasks, fixed separately in #19659), or - `native_runner_source_semantic_reconciliation_required` with a predecessor `paperclip_call_assigned_tool` call still pending (one task). The same-run retry then opened a replacement session. `archiveContinuityState` failed with `runner_continuity_break_archive_failed` and no detail. The third attempt failed with `runner_remote_state_unavailable: exit=1`, and the task was blocked with "Workspace repair required". Steps to reproduce 1. Save a native session whose checkpoint policy is `allow_replacement_after_resume_failure`. 2. Make the resume fail with a refusal that does not change state (an attach refusal, or a pending tool call). 3. Same-run recovery opens a replacement session and archives the saved continuity state. Expected behavior A deterministic refusal stops the run for an operator. The saved state stays intact. A run with a pending side-effecting call never reaches replacement. Transport drops keep the existing retry behavior. Root cause `executeNativeSession` checked only the checkpoint's replacement policy. The closed failure class (#19334) only had an effect when replacement was forbidden. Attach refusals and pending tool calls had no class at all, and the source-reconciliation refusal was a plain `Error`. ## What Changed - Runner (`provider_backend.rs`, `durable/mod.rs`): `run.attach` refusals that would change the durable Codex profile or a protected launch argument carry the closed classification `provider_attach_refused`. An attach blocked by `durable_pending_tool_calls` carries `provider_side_effect_pending`. These refusals already leave the saved state unchanged. - Contracts (`harness-driver.ts`): new closed classes `PROVIDER_ATTACH_REFUSED` and `PROVIDER_SIDE_EFFECT_PENDING`. A typed `NativeRunnerSourceReconciliationRequiredError` replaces the plain `Error` at the five `native_runner_source_*_reconciliation_required` throw sites. `runnerRecoveryFailureClass` reads the authenticated command-result classification or the typed error, never message text. - Runtime (`native-session-runtime.ts`): after either class, recovery never opens a replacement session, whatever the checkpoint policy. - An attach refusal ends with the existing `NativeProviderStateIncompatibleError`, which maps to the existing operator path `native_provider_state_incompatible`. - A pending side effect ends with the new `NativeProviderSideEffectPendingError`. - Server (`native-session-executor.ts`, `native-session-resume.ts`, shared `EXECUTION_RECONCILIATION_CAUSES`): `native_provider_side_effect_pending` is a permanent failure and an execution-reconciliation cause, so the board gets a reconciliation (record whether the action ran) instead of a generic Retry. The reconciled continuation does not rebind the saved session, which still refuses attachment. It starts a fresh provider session in the same workspace. The saved session and workspace are unchanged. - Archive (`archiveContinuityState`): a failed remote move now records the bounded (512 characters), redacted exit status and output in `runner_continuity_break_archive_failed: exit=… timedOut=… <output>`. The remote move already ran before the local move on `master`, so a failed remote move leaves the local state in place. `restart-interrupted-turn.ts` accepts the detailed message for its reattachment-loss retry. - Docs: `doc/execution-semantics.md` and `packages/paperclip-runner/docs/durable-recovery.md` describe the new rule. ## Verification Red before the fix, green after (each test first run against unchanged source from `origin/master` `644ee66f5d`): - `native-session-runtime.test.ts`, test "never replaces a session after a deterministic, non-destructive recovery refusal". Red: replacement was attempted (`TypeError: Invalid value used as weak map key` from the stub replacement). Green: typed error, and `openReplacementSession`, `openSession` and `openRun` are not called. - `harness-driver-backend.test.ts`, test "keeps the runner's closed class of a failed recovery". Red: the typed error did not exist. Green: both new classes are read from the command result and from the typed error. The same message text without the type stays unclassified. - `tests/codex_provider.rs` (real durable executor and fake Codex app-server): - `attach_with_a_pending_tool_call_is_classified_as_a_pending_side_effect`. Red: `left: None, right: Some("provider_side_effect_pending")`. - `saved_codex_thread_attaches_after_a_release_changes_its_feature_toggles` (protected sandbox change). Red: `left: None, right: Some("provider_attach_refused")`. - `native-session-executor.test.ts`: - Test "stops same-run retries for an operator while a saved side-effecting call has no result". Red: `expected 'native_session_interrupted' to be 'native_provider_side_effect_pending'`. - Test "retains the local controller checkpoint until remote continuity archiving succeeds: failed/timeout". Red: the message had no detail. Green: the message has the exit status and output, a token in the output is redacted, and the local runner and control-plane files are intact. Safety and regression checks: - New test "still replaces a session after an unclassified resume failure when the checkpoint allows it". It passes before and after: a transport drop still reaches replacement. - Existing test "types a runner-proven incompatible session when same-session recovery is required" still passes. - `native-session-resume.test.ts`: a source stopped with `native_provider_side_effect_pending` is not rebound. - `run-completion.test.ts` "replays saved user intent through normal Retry …" passes with the detailed archive message. - Runner Vitest: `native-session-runtime.test.ts` and `harness-driver-backend.test.ts` (266 passed); `runnerd-codex-transport-warm-attachment.test.ts` and `codex-app-server-driver.recovery.test.ts` (74 passed). - Rust: `--lib` (560 passed, plus one ACPX timing test that failed under load and passed when rerun alone), `codex_provider` (118 passed), `native_provider_backend` (45 passed), `durable_recovery` (3 passed). Not run: a live staging recovery at this revision. The local server typecheck could not resolve the unbuilt runner package, so required CI provides typecheck. ## Risks - A run that a replacement session would have continued after an attach refusal now stops for an operator. The refusal repeats on every retry, so the replacement was the only way past it. It also destroyed saved state, which is what blocked the staging tasks. A board Retry still continues the task in a fresh session in the same workspace. This is the existing `native_provider_state_incompatible` behavior. - A new failure code `native_provider_side_effect_pending` appears in run records and recovery actions. It joins `EXECUTION_RECONCILIATION_CAUSES`, so the UI shows the reconciliation flow, not Retry. - An attach refusal used to reach an automatic replacement; it now needs one board Retry. This is intended: the replacement archived saved state and failed part way on staging. - The pending-side-effect class means "no result delivered to the runner". The controller replays a recorded tool outcome only after attach. A call whose outcome the server already recorded therefore also stops for an operator. That is conservative; the reconciliation shows the recorded outcome. - Only a pending tool call classifies an attach blocker. Other blockers (active turn, ambiguous turn start, active goal) keep their current behavior. - The archive error message is longer. The one exact-match consumer now accepts the `code: detail` form. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edit, browser reads of staging). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 23:25:54 -06:00
Nicky Leach	b21884bdcf	Release and resume retained plugin Runner sessions in Core (#19611 )	2026-11-06 20:45:02 -08:00
DottaandClaude Opus 5.5	4612fa9fb4	fix(native): restart a drained Daytona run whose runner opened its session (#19711 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs on Daytona start a `paperclip-runnerd` process in a sandbox. A graceful shutdown (every rolling deploy) suspends live native runs "without cancelling provider work". The next boot then decides how each run continues. > - #19583, #19616 and #19684 let restart recovery start a remote bootstrap again when durable evidence proves that no provider work exists. > - A chaos replay found another window. The drain suspended a remote run just after its sandbox runner opened the provider session, before the controller sent a turn and before a session snapshot existed. > - Recovery accepted only an unfinished `session.open` in the controller checkpoint, so it blocked the task as ambiguous. A Retry recovered it, but automatic continuation was lost. > - A related replay showed that the same-run bootstrap reuses the sandbox. The interrupted runner can still hold the runner ingress port there (`runner_ingress_bind_conflict`). > - This pull request proves the "session opened, no turn" case and stops the interrupted runner before relaunch. > - The benefit is that a deploy during Daytona startup continues automatically, without a port conflict. ## Linked Issues or Issue Description What happened? A chaos replay (ACPX Claude on Daytona) sent SIGTERM while a remote run was starting. The drain recorded `restart_suspended`. The dying controller still launched the runner and finished `session.open`. On the next boot, restart recovery returned `ambiguous_remote_recovery_evidence` with disposition `blocked`. The decrypted controller checkpoint of that run showed: - `run.prepare` completed and `session.open` completed (provider session opened). - Committed events `session.started`, `session.capabilities.updated` and `session.goal.snapshot` (`ackedSourceSeq: 3`). - No turn command. No run-profile session snapshot and no projected provider events. On the next run in that replay, a same-run bootstrap relaunched the runner in the same sandbox (one lease for all attempts). It failed with `runner_ingress_bind_conflict: failed to bind listener: Address already in use`, because the interrupted runner from 51 s earlier was still listening there. The bounded same-run retry recovered it on its last attempt. Expected behavior When the durable controller record proves that no turn was ever sent, recovery starts the bootstrap again. It first stops the interrupted runner in the reused sandbox. Steps to reproduce 1. Start a native run on a Daytona environment. 2. Send SIGTERM to the control plane while the runner opens its provider session. 3. Restart the control plane after the controller lease expires. 4. See that restart recovery blocks the task with `ambiguous_remote_recovery_evidence`. Root cause: in `server/src/services/native-runtime/native-restart-recovery.ts`, the remote checkpoint proof (`remoteBootstrapProvenIncomplete`, #19583) required an unfinished `session.open` with no committed events, and no provider events at all. A completed `session.open` with only session-open events had no rule, so it fell through to `ambiguous_remote_recovery_evidence`. Refs #19583, #19616, #19684, #19524 ## What Changed - `native-bootstrap-proof.ts`: new `runnerdStateHasNoProviderTurn` proof. The state must hold only `run.prepare` and `session.open` (pending or completed) and no warm, staged or adopted state. Every committed event must be a session-open event, with contiguous sequence numbers from 1, so no event is hidden. Only the controller issues turn commands, and it writes each one to Postgres before sending it. - `native-runner-checkpoint.ts`: `nativeRunnerCheckpointRemoteBootstrapProof` replaces `nativeRunnerCheckpointProvesIncompleteBootstrap`. It also reads a bounded number of `event:N` parts and returns which proof holds. It still rebuilds the state in memory. - `native-restart-recovery.ts`: a remote run can now claim `bootstrap_incomplete` with reason `remote_session_opened_without_turn`. Projected provider events may only be session-open events (`harness.ready`, `session.started`, `session.resumed`, `session.updated`), and only together with the stronger session-opened proof. A turn or provider event still blocks. - `native-session-executor.ts`: - For a remote sandbox target, the executor accepts the same proof on the restored record. - Before the same-run bootstrap relaunches, `stopStaleRemoteRunnerBeforeBootstrap` stops the interrupted runner in the reused sandbox. It uses the verified identity marker and process check from reattachment, which signals only a process whose identity matches this run: SIGTERM, a wait of up to 10 s, then SIGKILL. - The stop is best effort. If no prior runner can be verified, nothing is stopped, and the bounded same-run retry still handles any conflict. - `doc/run-log-events.md`: document the reason and the stop. ## Verification - Red, then green: `cd server && npx vitest run src/services/native-runtime/native-runner-restart-recovery.integration.test.ts -t "drained remote"`. Before the fix, the opened-session case (with a projected `session.started`) returned `blocked` / `ambiguous_remote_recovery_evidence`. - These keep the hold: a `turn.start` command, a committed `turn.started`, an acknowledged event that is missing from the record, and a projected `turn.started`. The executor test restores the checkpoint and checks that the local proof is false and the remote proof is true. - Regression: `native-runner-restart-recovery.integration`, `native-session-executor`, `native-restart-recovery`, `native-bootstrap-retry.integration` and `heartbeat/recovery`: 1460 passed, 14 skipped. - Stale runner stop: controlled sandbox-runner tests (`native-session-executor.test.ts`, "stale runner stop before a same-run remote bootstrap") run the real reattachment identity and process scripts against a fake sandbox runner: - a verified runner that exits on SIGTERM gets SIGTERM only; - a runner that ignores SIGTERM gets SIGTERM, then SIGKILL; - a runner with another identity gets no signal; - no live prior runner means no signal. - Live Daytona proof (wave `w-20261107T021837-s29`, ACPX Claude, `--faults K2 --triggers time,result`, 4 tasks, concurrency 1). The image was built from current master (`paperclip-daytona-runner@sha256:71f711ab…`), and its content id matches this branch. - All 4 tasks crossed a graceful restart, and all passed R1. Two of the K2s fired on the `result` trigger, during a running remote turn. - The K2 during a turn reattached to the live remote runner (`reattach_remote_runner`, generation 2), as designed. - Task t01 failed R6 with an orphan Daytona sandbox that no lease references. The graceful restart came during sandbox creation. This is a separate signature, outside this change. - The timed faults did not land in the exact "session opened, no turn" window. The integration tests cover that window with the evidence from the saved replay database. ## Risks - The proof relies on two facts: only the controller issues turn commands, and the controller writes each command before sending it (`#sendJson` waits for the checkpoint flush). A turn always leaves a `turn.start` command in the record. - The provider session opened in the old sandbox runner is not resumed. The new bootstrap opens a fresh session. No turn ran in the old one, so no work is lost. - The runner stop sends signals inside the run's own sandbox. It signals only after the existing reattachment identity and process checks pass. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, GitHub CLI) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 20:58:32 -06:00
Dotta Paperclip Codex	18be7ac696	fix(server): retire idle native sessions before tenant sleep (#18988 ) ## Thinking Path > - Paperclip lets people manage AI agents and their work. > - Native agents can retain a warm session between useful turns. > - That owner must close and save its files before the tenant can sleep. > - A failed close must keep the owner visible until retirement succeeds. > - This PR prepares idle resources under the exact current admission hold, then takes a fresh safety snapshot. > - Busy turns, uncertain closure and changed holds still prevent sleep. > - CI also exposed a chat authorization race. The fix repeats full authorization after the exact delivery commits. ## Linked Issues or Issue Description Refs #18639 for local work and incoming wake boundaries. Related #18829 covers suspension receipts after admission. What happened? Idle preflight did not retire warm Native owners. Three failed-close paths could forget an owner without proving that its provider stopped. CI also found a false chat rejection when a delivery changed from processing to processed between two validation reads. Expected behavior Retire only idle owners under the exact bounded hold. Wait for closure and file collection. Retain failed owners. Scan safety after preparation. Keep diagnostic reads passive. A newly committed chat delivery must pass the full current authorization checks before attestation succeeds. Steps to reproduce Finish a warm Native turn. Start an idle drain and request safety with its owner ID. Delay or reject provider closure, or replace the hold during preparation or scanning. The report must keep the instance awake. For the chat race, commit the exact delivery between the first binding read and its fallback read. Paperclip version or commit Current head `7f6f02b611dcf5a9a2b05173a44e9631eaf78058` normally integrates actual master `1aa0c3639714f4976cfa56f95aae6e2b7d1235ac`. The merge preserves every unrelated master path. Fresh exact-head CI and complete review remain required. ## What Changed - Retain uncertain Native owners after failed restart, idle timeout and failed-turn closure. - Prepare Native retirement through the existing heartbeat facade. Reuse provider closure and stopped-file collection. - Track preparation under the exact idle owner and expiry. Reject new activity, changed holds and expired holds before and after awaited work. - Take the final durable and local safety snapshot after preparation. Keep diagnostic inspection passive. - Cover ephemeral workspace retirement and useful next use, supported plugin drain and SDK persistence, busy owners, failed closure and hold replacement. - Keep dedicated recovery storage explicit. Use the same validated options for support and storage. Probe-free helpers retain the original implementation bodies and preserve default callers. - Complete the Native fixture through real retirement and budget evaluation. Keep its rejection, file, credential, accounting and strict NONE assertions. - Repeat the complete chat authorization once if the fallback sees the exact batch now processed. Preserve owner locks, principal and binding checks, denial handling and the original retry deadline. ## Verification - The support-mode regression reproduces one failure with four passes under the old policy, then passes all five controls. It retains prior allocation and cleanup cases. The explicit mode does not probe embedded PostgreSQL or use an admin alias. - The deterministic chat race baseline reproduces two authorized failures with all four current-authority denial controls passing. Both Slack and Discord use real PostgreSQL and the real authorization function. The negative setup commits before the attestation transaction; it does not claim to revoke access through a row lock. - The product fix passes normal own App declaration emission with no public declaration changes, plus both current normal owning compiler programs (399 and 611 roots). The same six controls pass after the fix. The full original file plus these controls passes all 30 tests with no skips. - The original controlled Native continuation passes: one test, no failures or skips, 1.718 seconds in its body and 23.138 seconds in its launcher. The original 40-second case and 180-second outer bounds are unchanged. It verifies held closure, strict NONE, an ordinary authorized comment, saved session/files, useful output, encrypted credentials, accepted results and settled accounting. All ten observed processes and groups close; worker connections and listeners reach zero. A separate post-owner audit finds zero sessions; the successful owned database is normally removed. - Earlier preparation, HTTP, lifecycle, workspace and plugin controls remain source-specific evidence. They are not a claim that every final-tree test ran locally. - The original CI assertion log is retained. The controlled race proves a possible cause; the original CI SQL schedule was not captured. Two other jobs lost their self-hosted runners and provided no test verdict. Earlier asset timeouts, readiness failures and unexplained later recovery remain separately recorded. No timeout, failure predicate or authority check was weakened. Focused public commands: ```sh pnpm exec vitest run server/src/__tests__/idle-sleep-preparation.test.ts server/src/__tests__/instance-settings-routes.test.ts --maxWorkers=1 --no-file-parallelism pnpm exec vitest run server/src/__tests__/heartbeat-reviewed-chat-binding.integration.test.ts --maxWorkers=1 --no-file-parallelism ``` Local database qualification uses fresh owned test databases on existing PostgreSQL, bounded pools and one worker. The local allocation adapter changes only database allocation for this test module; it runs normal migrations and client cleanup. It is not part of the published source. No new cluster is created. Local acceptance measured the `0a69efa6` integration with the same committed authorization bytes. The later normal master merge adds startup-timeout classification and recovery changes. Those paths are outside the controlled healthy-backend witness; retirement and authorization hunks are preserved. Exact-head remote CI at `7f6f02b611dcf5a9a2b05173a44e9631eaf78058` has completed with 56 successful contexts and four skips. The complete current Greptile summary is5/5 with no new actionable findings; all review threads are resolved. Ponytail’s current turn is explicitly paused without a current-head verdict; earlier actual findings are fixed and accounted for. No whole final-tree local repository build/test run, live-provider restore or deployed suspension is claimed. Actual master subsequently advanced to `d5c4f7dd02d408a261cedd9a1926b9900a4aedc4`. Its local normal-merge projection with this reviewed head preserves all20 owned paths and every unrelated master blob and mode. Current Runner TypeScript, App declarations and their full owning compiler programs pass. Focused Native cleanup and warm-continuation composition checks are being completed before merge; the new upstream suspension-retry behavior is not inferred from the older controlled-backend witness. The remote PR head remains unchanged. ## Risks - Privileged idle preparation can close idle warm Native sessions. It requires the exact current hold. - Failed closure retains its owner and can keep the tenant awake until normal retry succeeds. - A changed hold cannot authorize sleep. Started cleanup remains tracked until it settles. - The chat repair can perform one extra full authorization read. It keeps the existing deadline and does not grant authority from the fallback query. - Unsupported resources and active work retain their existing blockers. This PR does not certify every transport, plugin or deployed host. ## Model Used OpenAI `gpt-6.1-sol` produced the delegated changes and verification. OpenAI Codex, based on GPT-6, performed managing review and integration. Both used repository tools and code execution. The coordinator's exact backend snapshot and context-window limits were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green at exact head `7f6f02b611dcf5a9a2b05173a44e9631eaf78058` - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups at the exact current head - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Codex <noreply@openai.com>	2026-11-06 20:49:30 -06:00
DottaandClaude Opus 5.5	30857db55e	fix(native): recover a run killed between runner launch and its identity record (#19694 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs start a local `paperclip-runnerd` process. The server records the request to start it, then records its process identity after it spawns. > - If the server is killed between those two records, there is no process ID to check after the restart > - Restart recovery then restarted the bootstrap, but the executor failed with `durable_identity_unreadable` because the runner never wrote its state > - The user's Retry then waited forever with "Retry is waiting for Paperclip to verify that the previous execution stopped", because the stop check needs a process ID > - This pull request restarts that bootstrap and proves on the host that no runner for the run exists > - The benefit is that a restart in the runner launch window no longer loses the task or leaves Retry permanently stuck ## Linked Issues or Issue Description What happened? In restart-chaos testing, an ACPX Claude run was created and the server was SIGKILLed 1.3 s later, right after `native.process_start_requested` and before the runner identity was recorded. On restart, recovery claimed `bootstrap_incomplete` (`dead_runner_before_provider_identity`). The executor then threw `runner_state_identity_mismatch: durable_identity_unreadable`: the state root existed, but `control-plane/control-plane-state.json` had never been written. The run failed with `native_continuation_requires_reconciliation`. A Retry was accepted (HTTP 200), but no run was ever created. The issue stayed blocked with `retryWait.reason=process_identity_missing` for over 20 minutes and through another restart. Expected behavior When the runner never wrote state, no provider started, so the bootstrap can start again. A Retry should be admitted once the host proves that no runner for the run exists. Paperclip version or commit `master` at `fb3cf1a072`. Steps to reproduce Create a native local task and SIGKILL the server about 1 s after the run starts, after the runner launch is requested and before its identity is recorded. Restart the server, then click Retry. ## What Changed - `migrateRunnerdStateRootForExecution` (`native-session-executor.ts`): for a `bootstrap_incomplete` restart on a local target, a root is quarantined (kept for inspection) and the bootstrap starts again only when both of these hold: - the root has no `control-plane-state.json`; - no live runner for the run exists on this host. A state file that exists but cannot be read, or a runner that started but has not connected yet, still fails closed with `durable_identity_unreadable`. - `unidentifiedLocalRunnerProvenAbsent` (`native-session-executor.ts`) proves that a runner is absent when both of these hold: - no host process carries the run's `--run-id <runId>` argument (`paperclip-runnerd` is always started with it, detached and unreferenced); - no saved control-plane state for this run exists, in the canonical root or in quarantine. A provider starts only after its runner connects and writes that state. - `native.process_start_requested` now records the launching `hostname`. - `recordUnidentifiedLocalRunnerStop` (`native-local-process-stop.ts`) applies the proof only when all of these hold: - the run is native; - the run has no process ID; - its latest launch event is `native.process_start_requested`, from this host; - all of its leases are local. A launch recorded without a host stays blocked. - The process listing runs asynchronously with a 5 s deadline. A failed or slow listing proves nothing. It then writes `native.local_process_stopped` with null process IDs and `basis: "unidentified_runner_absent"`. A later launch request invalidates it as usual. - The continuation and Retry gate (`explicit-native-continuation.ts`) uses this proof before returning `process_identity_missing`. A dry run checks the proof but records nothing. - `doc/run-log-events.md` documents the new receipt basis. ## Verification - `native-session-executor.test.ts`: - The bootstrap restart with absent state was red before the fix (`durable_identity_unreadable`) and is green after it (root quarantined, backend admitted). - Unreadable state, and a live runner that has not connected, still reject. - Proof cases: absent → true; a live process with `--run-id <runId>` → false; this run's saved state → false; another run ID → false. - `explicit-native-continuation.test.ts` (embedded Postgres, real admission): - The `absent` and `dry_run` cases were red before the fix (blocked) and are green after it (admitted). Only the real admission records the receipt. - The `live_runner`, `saved_state`, `foreign_host` and `unbound_launch` cases stay blocked. - Broader local run after the review fixes: `explicit-native-continuation`, `native-session-executor`, `native-recovered-owner-followup`, `native-runner-restart-recovery.integration` and `task-retry` gave 1836 passed, 0 failed. - A chaos replay of the original task (`--replay w-20261106T232854-s7103/t26`) passed (R1 PASS). Its kill landed after the runner identity was recorded (`resume_dead_runner`), not in the launch window, so the tests above are what reproduce that window. - `tsc --noEmit` for `server` passes. ## Risks - The proof depends on `ps -axww` listing every process on the launching host. If `ps` fails or times out, the proof fails closed (still blocked). Remote runners and other hosts are excluded. - A runner that was spawned but has not yet written its state could be mid-start when admission runs. It would still carry `--run-id` in its arguments, so the proof refuses. - Only the bootstrap path with no state file changes. Ownership, checkpoint and reattach checks are unchanged. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test execution). Extended reasoning enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 20:06:36 -06:00
DottaandClaude Opus 5.5	9a52a2dfeb	fix(native): retry ACPX session.open startup timeouts and give cold Claude a 60 s bound (#19661 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runs start an ACPX provider through Runner. `session.open` opens the provider session before any prompt. > - On a loaded host, a cold ACPX Claude `session.open` took longer than its 30-second bound. The controller reported `runner_local_connect_failed: PRP command session.open timed out`. > - The server classified this as the generic `native_session_interrupted` with `ambiguous_state`, so the task was blocked at once. A person had to click Retry, sometimes several times. In one restart-chaos storm, one task needed three Retry clicks, and on another task both Retry runs timed out too. > - The server already has a bounded same-run bootstrap retry for failures that come before provider work (#18854). It did not recognize this timeout. The 30-second bound also cut off normal slow startups. > - This pull request gives cold ACPX Claude a 60-second `session.open` bound, like Pi and OpenCode (#19487). It also retries a startup `session.open` timeout automatically on the same run, from clean state. > - The benefit is that a slow cold start on a busy host completes, or retries itself, instead of waiting for a person. ## Linked Issues or Issue Description Refs #18854 (bounded bootstrap retry for definite `session.open` failures), #19487 (60-second cold OpenCode admission), #19665 (the bootstrap proof that this change reuses). What happened? ACPX Claude runs failed with `runner_local_connect_failed: PRP command session.open timed out` about 35 to 40 seconds after they started. The coordinator recorded `native_session_interrupted` with `recoveryMode: "ambiguous_state"` and `providerSessionEstablished: false`. The task was blocked with a board-owned action. No provider event and no checkpoint existed. The host 1-minute load was 17 to 44, sometimes with no restart nearby. Expected behavior A normal slow cold start completes. A `session.open` that still times out during runner startup, with no provider event, no checkpoint and no turn, retries on the same run with backoff. At most two automatic attempts follow the first. A run where provider work may have started is never retried automatically. Steps to reproduce 1. Run ACPX Claude tasks on a host with a 1-minute load of about 25 or more. 2. Observe a run fail after about 35 to 40 seconds with `PRP command session.open timed out`. 3. Observe the task blocked with `ambiguous_state` and a manual Retry. Paperclip version or commit `master` at `6c7cc2c5d6`. ## What Changed The bound (evidence below): - `coldAdmissionTimeoutMs` gives cold ACPX Claude 60 seconds (live adoption keeps 30 seconds). The runner's Rust sidecar `session_open_timeout` and the sidecar's provider handshake (`acpxSessionHandshakeTimeoutMs`) give Claude and Pi 60 seconds too. All three bounds must agree, or the shortest one fails first. Claude's warm run attachment uses the same cold budget, like Pi. The automatic retry: - `nativeSessionFailureSourceCode` returns `provider_session_open_timeout` only for exactly `<startup failure code>: PRP command session.open timed out`. The startup failure codes are `runner_local_connect_failed`, `runner_direct_wss_failed` and `runner_ingress_unavailable`. The transport uses them only before startup completes. After startup the code is `provider_transport_failed`, which is unchanged. - `nativeProviderRecoveryEvidence` treats the cause as pre-session. Durable evidence still decides: a bootstrap retry needs no provider event (`session.started`, `turn.started` and the others) and no checkpoint. The existing budget applies: three attempts with backoff of about 30 seconds, then about 60 seconds. When the budget is used, the task is blocked with the existing "Provider startup failed on 3 attempts before any provider work began" action. - A timeout is indeterminate: the runner can open a provider session after the controller stops waiting. Live runs showed both leftovers, so the retry now starts from clean state: - `prepareNativeSessionOpenTimeoutRetry` (server) runs when the failure settles. It quarantines a local runner root only when the restart-recovery proof (`runnerdStateProvesIncompleteBootstrap`, #19665) shows that no turn began and every recorded process is gone. Any other root, and any remote runner, stays `ambiguous_state` as before. - `resetAbandonedBootstrapRecord` (runner ACPX adapter) runs on a fresh open (no runner-recorded identity). If ACPX has a saved record for the session key with no messages, no prompt and no events, it is marked `reset_on_next_ensure`. Claude saves nothing before the first prompt, so reusing such a record fails the first turn with `session_resume_required`. A record that holds any conversation is unchanged. - A chat retry of an exhausted `provider_session_open_timeout` run is accepted like the other pre-provider causes. - `doc/execution-semantics.md` and the Runner README describe the cause and the bound. ## Verification Measured startup distribution (ACPX Claude `runner.session.bootstrap` / `runner.session.startup` spans from restart-chaos run events, 1-minute load 8 to 44): \| Source \| Bound \| Successful cold bootstraps (s) \| Failures \| \| --- \| --- \| --- \| --- \| \| Earlier storm evidence \| 30 s \| 2.9, 4.7, 7.4, 8.6, 8.7, 26.2, 27.4, 27.6, 29.2 \| startups ended at 34.9 to 40.3 s (the 30 s `session.open` bound plus spawn and `run.prepare`) \| \| This branch, wave s3601 \| 60 s \| 6.3, 7.8, 8.1, 13.4, 14.4, 22.3, 22.8, 22.9, 37.3 \| 0 `session.open` timeouts \| \| This branch, wave s3602 \| 60 s \| 13 samples, p50 10.5, p90 17.9, max 25.7 \| 0 `session.open` timeouts \| \| This branch, wave s3610 (final head, load 28 to 42) \| 60 s \| 10.4, 11.3, 11.7, 12.0, 13.3, 14.2, 29.6, 30.4, 30.9 \| 0 `session.open` timeouts \| Under load, successes clustered just under 30 s and failures sat just above it, so the 30 s bound was cutting the distribution off. With 60 s, cold starts of 30.4, 30.9 and 37.3 s succeeded that the old bound would have failed. 60 s is about 1.6 times the largest observed success, and it matches Pi and OpenCode. Live restart-chaos waves (local, ACPX Claude only, 4 in flight, `--rc06-masked`, real provider, K1/K2 restarts): - s3601 (8 tasks, load 19 to 32): 7/8 PASS. No `session.open` timeout. The one failure is an R3 Retry signature after K1 (`native_restart_recovery_blocked`), not this cause. - s3602 (8 tasks, load 17 to 39): 8/8 PASS. No `session.open` timeout. Three Retry clicks were for `native_provider_terminal_failed` (`provider_turn_lost_on_restore`), a different restart signature. - Forced timeout, s3609 (4 tasks, B5, K2): a local, uncommitted fault injection gave every other cold startup a 1.5 s `session.open` deadline. Result 4/4 PASS. Run 4fe6537a timed out, retried itself on the same run, and then succeeded through four K2 restarts with no person involved. Run 02799a85 hit the injection three times in a row; the budget stopped it after attempt 3, and one Retry completed it. - Before the two clean-state fixes, the same forced wave (s3607 and s3608) showed the retry reusing leftover state: `turn.start` failed with `session_resume_required`, and then "ACPX session is not open". Those runs are the red evidence for the two clean-state fixes. - Final head, s3610 (8 tasks, K1/K2, no injection, load 28 to 42): 8/8 PASS, 0 `session.open` timeouts. The one Retry click (t06) was for `provider_turn_lost_on_restore`, a different restart signature. Tests: - Red before the fix: `native-startup-retry.integration.test.ts -t timeout` (embedded PostgreSQL, real executor, real resume claim). The 3 retry cases failed with `native_session_interrupted` / `ambiguous_state`. - Green: `native-startup-retry.integration.test.ts` passes 36/36. New cases: - for each startup failure code, a timeout retries on the same run with backoff, keeps the task `in_progress` with an agent-owned `resume_native_run` wake, and blocks after attempt 3 with a board-owned action; - no automatic retry when `session.started` or `turn.started` is durable, when the code is `provider_transport_failed`, or when another command timed out; - a real runner root with an opened session and no turn is quarantined; a root with a turn command stays `ambiguous_state` and intact; a remote runner is never prepared. - `codex-runtime-adapter.test.ts`: 113 pass. New cases: a 45 s delayed handshake is admitted for Claude and rejected for Codex; a never-prompted record is reset only on a fresh open; records with messages, a prompt or events are kept. - `abandoned-bootstrap-record.test.ts` (real `createAcpRuntime` with `createRuntimeStore` and a controlled ACP peer that, like Claude, cannot load a never-prompted session): without the reset, a fresh open reuses the record and the first turn fails with SESSION_RESUME_REQUIRED after `session/load`. With the reset, the provider receives `session/new` and the turn completes. - `vitest run src/drivers/acpx src/cli` in the runner: 107 files, 1,935 tests pass. - `native-session-executor.test.ts -t "session.open\|startup\|bootstrap"`: 144 pass. `runnerd-codex-transport.test.ts -t "cold process admission"`: 4 pass. `cargo test -p paperclip-runner-core --lib admission_budget` passes. - `npx tsc --noEmit -p server` and the runner `tsc -p tsconfig.test.json` pass. ## Risks - This changes Runner Rust code. The Daytona runner image must be rebuilt after merge for remote sandboxes to get the 60-second sidecar bound. Until then, a remote Claude `session.open` still fails at 30 seconds in the sidecar (as a definite failure, which already retries). A remote timeout stays operator-owned, because a remote root cannot be proven clean here. - A cold ACPX Claude start that hangs now fails after about 60 seconds instead of 30 seconds. - A timed-out `session.open` may have created an empty provider session. No prompt was sent. The retry quarantines that runner root and asks ACPX for a new session. - Same-run retries of a fresh-attempt Retry run can still fail with `isolated_task_directory_binding_mismatch`. This is RC-35 (workspace re-preparation), which has its own fix in progress. - No schema or API change. The new error code appears on runs and in run error codes. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, file edits, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 19:37:11 -06:00
DottaandClaude Opus 5.5	6c7cc2c5d6	fix(native): do not spend a recovery attempt resuming a cleanly drained run (#19625 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs have an execution recovery budget of three provider attempts, so a run that keeps failing stops retrying > - A graceful shutdown (deploy or SIGTERM) drains native runs: it suspends them for reattachment by the next server > - When the runner did not survive, the next boot resumed the turn and charged one attempt > - Restarts that the run did not cause used up its budget, so recovery stopped and the task needed a manual Retry > - This pull request does not charge an attempt for resuming a run that a graceful drain suspended > - The benefit is that deploys and graceful restarts no longer exhaust a healthy run's recovery budget ## Linked Issues or Issue Description What happened? In restart-chaos testing, native runs reached `execution_recovery_budget_exhausted` (3/3) only because of server restarts. In one wave, two restarts crossed each of two runs. In a second schedule (K1, then K2 when ready, then K2 again), each drain correctly suspended the local ACPX Claude run, yet the fourth boot blocked it with `native_restart_recovery_blocked`. Every resume after a drain was classified `resume_dead_runner` and charged an attempt (`nextNativeProviderAttempt` in `native-restart-recovery.ts`, applied by the executor's execution claim). Expected behavior A restart that the run did not cause should not consume its recovery budget. A graceful drain records `restart_suspended` and `awaiting_runner_reattach`. The resume that follows should keep the attempt count. Paperclip version or commit `master` at `4ef3fa4271`. Steps to reproduce Start a native local run. Send the server SIGTERM three times while the run is active, with a full restart each time. The third resume is blocked with `execution_recovery_budget_exhausted`. ## What Changed - `suspendedByCleanDrain()` in `native-restart-recovery.ts` recognizes a run whose most recent recovery claim is older than the drain's `restart_suspended` entry. Ownership waits and deferred blocks between the drain and the next claim do not erase it. The next claim consumes it. - A scan that waits for the old controller to exit (`awaiting_evidence`) records one history entry per reason, so repeated scans cannot push the drain record out of the bounded history. - Restart claims carry `chargesProviderAttempt`. It is `false` for a claim that follows a clean drain. - `nextNativeProviderAttempt()` takes that flag. The executor passes it when it takes the execution lease, so a drained resume keeps the current attempt. - The first execution is always counted: a run drained at attempt 0 (no execution claimed it yet) still charges its first provider start. - A run drained during its third attempt may continue that attempt. The local resume and remote bootstrap gates skip the `>= 3` exhausted check only for a non-charging resume, and the executor still refuses a fourth attempt. - A crash restart (no drain record) still charges one attempt. Reattachment to a live runner still charges none. - `doc/execution-semantics.md` states the rule next to the three-attempt budget. ## Verification - New test in `server/src/__tests__/heartbeat-native-restart-suspension.test.ts` (embedded Postgres, real heartbeat drain and restart recovery): - Red before the fix: the claim after the drain had no `chargesProviderAttempt: false`. - Green after the fix, for drained attempts 0, 1 and 3: the crash claim charges (`true`). After the drain, a controlled backend lets the executor take its execution lease and records the saved `native_run_finalizations.attempt` at provider dispatch: 1, 1 and 3. Attempt 3 is claimed instead of blocked. - Red at the executor: when the executor ignores the flag, the drained attempt-1 resume saves 2. - Ownership-wait case: the drained controller (the test process) is still alive, so two scans wait. The history keeps `restart_suspended` followed by one `awaiting_evidence` entry. After the controller is gone, the claim does not charge, and attempt 3 continues. - Classifier cases (local resume and remote bootstrap): attempt 3 after a crash is still `execution_recovery_budget_exhausted`. Both new cases were red on the previous revision. - `heartbeat-native-restart-suspension.test.ts` 12/12, `native-runner-restart-recovery.integration.test.ts` and `native-runner-ownership.test.ts` 94 passed. - `tsc --noEmit` for `server` passes. ## Risks - A run that itself makes the server shut down gracefully, again and again, would no longer be stopped by this budget. Graceful shutdowns are operator or deploy actions, so this is unlikely. A crash loop (SIGKILL, OOM) still charges an attempt on every restart, and a resumed turn that fails still goes through the existing failure and replacement budget. - This changes only the attempt count. Ownership checks, checkpoint verification and the exhausted gate are unchanged. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test execution). Extended reasoning enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 18:12:12 -06:00
DottaandClaude Opus 5.5	e89be6ee83	fix(native): re-bootstrap a runner that a graceful drain suspended during bootstrap (#19665 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs start a local `paperclip-runnerd` process. The control plane keeps a durable state root for each session. > - After a restart, recovery gives a run with no provider evidence the `bootstrap_incomplete` disposition. The executor quarantines the old root and starts the bootstrap again, but only when the root proves that no provider work started. > - #19524 made that proof accept a root killed after `run.prepare` and before `session.open` (a hard kill). > - A graceful restart (SIGTERM) leaves different shapes. The drain sends `runner.drain` and `runner.suspend`, and the kill can land just after `session.open` failed or just after it started a fresh session. The proof refused both shapes. The run failed with `runner_state_identity_mismatch: bootstrap_not_proven_incomplete`, although no turn was ever sent. > - This pull request extends the proof to both drain shapes and keeps provider work fail-closed. > - The benefit is that a graceful restart during native bootstrap continues automatically. ## Linked Issues or Issue Description Refs #19524 What happened? Restart chaos waves (ACPX Claude, K2 graceful restart about 10 s after the run started) failed with `adapter_failed` and `runner_state_identity_mismatch: bootstrap_not_proven_incomplete`. Startup recovery chose `bootstrap_incomplete`, and the executor proof refused the state root. Two real roots were kept: 1. `session.open` failed (the ACPX sidecar timed out), then `runner.drain` completed and `runner.suspend` completed. There was 1 connection, no committed or acknowledged event, an empty runner outbox, and `nextSourceSeq: 1`. 2. `session.open` completed with a freshly started session (`status: "started"`). The committed events were only `session.started`, `session.capabilities.updated` and `session.goal.snapshot`, and the runner outbox held one `harness.diagnostic` (the sidecar process). No turn command existed, and the sidecar process had exited. The database had not yet recorded the session events, so the classifier saw no provider evidence. Steps to reproduce 1. Start an ACPX Claude task on a local runner. 2. Send SIGTERM to the control plane about 10 s after the run starts, before the first turn. 3. Let the control plane restart. Expected behavior Recovery starts the bootstrap again, and the task continues without a user action. Actual behavior The run fails and the task stops. ## What Changed - `native-bootstrap-proof.ts`, pre-session proof: a `session.open` that is pending or `failed` is now accepted, as are the runner lifecycle commands `runner.drain`, `runner.suspend` and `runner.shutdown` in any state. The controller journal and the runner journal must both agree. Any committed or acknowledged event, any turn command, a completed `session.open`, and any queued runner event still fail closed. - New `runnerdStateIsOpenedSessionBeforeTurn`. It accepts a completed `session.open` only when all of these are true: - the session was freshly started (`status: "started"`, not resumed); - the commands are only `run.prepare`, `session.open` and runner maintenance commands; - every committed event and every event in the runner outbox is a session setup or runner lifecycle event; - `ackedSourceSeq` equals the count of committed events, with no gap; - the runner journal is present; - every recorded provider process ID has exited on this host. - `native-session-executor.ts`: the executor applies the opened-session proof only for a local execution target. Remote runners stay fail-closed for that shape, because their process IDs belong to another host. The remote checkpoint classifier still reads only commands 1 and 2. ## Verification - Red before the fix: "allows a fresh bootstrap after a drain suspended a runner whose session.open failed" and "allows a fresh bootstrap after a drain that landed between session.open and the first turn" both failed with `expected false to be true`. Green after the fix: the `native incomplete-bootstrap evidence` block passes 32/32. - Fail-closed counter-tests: - completed `session.open`; - committed event, acknowledged event, or queued runner event; - turn command, `turn.stop`, or a turn processed by the runner; - `turn.started` or `provider.event`; - resumed session; - compacted events; - no recorded provider process; - missing runner journal; - live provider process; - remote runner. - I ran the new proof read-only against the three real kept roots: the K1 root from #19524, the K2 `session.open` failed root and the K2 opened-session root. All three return `true` for a local runner. The opened-session root returns `false` when the processes are on another host. - `native-session-executor.test.ts` and `native-restart-recovery.test.ts` filtered to bootstrap, restart, remote and drain: 333 passed. `native-runner-restart-recovery.integration.test.ts` and `native-bootstrap-retry.integration.test.ts`: 106 passed. `tsc --noEmit -p server/tsconfig.json` passes. - Live restart chaos replay: see the PR comment. ## Risks - After a `session.open` failure or timeout, the provider sidecar might have created an empty session. No turn command was sent, so no provider work can repeat. - An opened session before any turn is discarded, and a new one is started. The proof requires a fresh `started` session, so no earlier conversation is lost. Resumed sessions stay fail-closed. - The opened-session proof is local only. It requires every recorded provider process to be gone, so a live provider process blocks it. - Out of scope: the Retry offer when the proof fails legitimately (RC-05/06). > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude Code with tool use (shell, file edits, test execution). Default context window. Extended reasoning was available. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 17:55:47 -06:00
DottaandPaperclip	9197abf673	fix(native): preserve controller state when remote archiving fails (#19663 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native continuity replacement must preserve the state needed to recover saved work. > - The current replacement moves local controller files before it knows whether the remote archive succeeded. > - A remote failure then leaves the local controller without its recovery files. > - This pull request confirms the remote archive before it moves those files. > - A user turn that never started can use the existing bounded Retry path after verified preservation and cleanup. > - The task retains its saved work and can continue once under current user authority. ## Linked Issues or Issue Description Refs #19638, #19547, #19399 and #19659. The prerequisite #19638 merged as `0819f4ea78dd37fba87341b314be9dac078c1410`. This PR now targets master. A normal dependency integration at `febed950f4` keeps only the six-file archive repair in the PR diff. Fresh master-based CI and review are required before merge. What happened? A native continuity replacement moved its local controller journal and provider checkpoint. The following remote archive failed. A later recovery attempt could no longer read the local recovery files. The first-attempt failure also had no normal Retry receipt for the saved user turn. Expected behavior A failed or timed-out remote archive retains the local recovery files and starts no replacement provider. A user turn that has not started can request one normal Retry. The existing user, provider-turn, Stop, preservation, cleanup, budget and approval gates remain in force. Steps to reproduce 1. Keep a local controller journal and provider checkpoint for a retained remote conversation. 2. Request continuity replacement and fail or time out the remote archive command. 3. Check the local controller files before any later recovery. 4. Settle the native failure under a current user wake and recover the saved workspace. ## What Changed - Confirm remote continuity archiving before moving the local controller state. - Add the exact remote archive failure to the shared bounded user Retry admission from #19638. - Verify failed, timed-out and successful archive ordering with real local files. - Verify ordinary failure settlement and saved-file recovery produce one fresh successor. Keep Stop, system wake and unsettled-provider holds. - Document the file preservation and Retry contract. ## Verification - RED: failed and timed-out remote archives removed the local controller files. Reading those files failed with ENOENT. The saved-user Retry request returned false instead of true. - Initial archive unit verification: eight cases passed, including the three archive outcomes and five existing Cloud archive safety cases. - Initial stacked revision: `9104ee399b`, based on #19638 at `d22d8f9d5ee3dc36550a94d55c1d8376f3b9aab2`. - Final master-based revision: `febed950f4`. All 13 affected cases passed across three Vitest files after dependency integration. They include the seven-mode real PostgreSQL completion case, saved-file successor and negative-hold cases, and real-file archive outcomes. All seven server typecheck layers passed again. Module boundary, whitespace and added-code-comment checks passed. All required CI passed on this exact head. The final snapshot has 59 successful checks, three intended skips, one neutral result, and no pending or failed checks. Current-head Greptile and the independent review are both 5/5, with no open findings. - Final local revision: `9104ee399b`, macOS ARM64, Node 26.4.0, pnpm 9.15.4 and embedded PostgreSQL. The three real-file archive outcome cases passed. The real PostgreSQL completion test passed all seven internal scenarios, including archive failure, prior provider activity, system wake, Stop before and after terminal settlement, and stale runtime promotion. Own Runner/shared/SDK builds and all seven server typecheck layers passed. Nine saved-work continuation and negative-hold cases passed. They verify exported file contents, one user-attributed fresh successor, idempotent hooks, no second automatic chain, and first-use manual Retry for held cases. - `node scripts/check-module-boundaries.mjs`, whitespace checks and the added-code-comment audit passed. - Focused commands from `server/`: Vitest with `-t 'retains the local controller checkpoint until remote continuity archiving succeeds'`; `-t 'replays saved user intent through normal Retry'`; and `-t 'continues a user turn once\|does not continue automatically'`. PostgreSQL cases used a fresh local cluster and a 120-second setup hook budget. Application timeouts were unchanged. - The initial stacked head passed all test, build, typecheck and E2E jobs. Greptile and the independent review both scored it 5/5 with no findings. These results do not replace the fresh master-based qualification now running. ## Risks - A remote command can fail after it changed remote state. This change retains local evidence in that case. It does not claim a distributed atomic transaction. - The exact archive error can request one normal Retry only under the existing saved-user admission. The normal path must prove stop, saved files and cleanup before a successor starts. - The protected launch-argument rejection remains enforced. Its exact hosted argument difference was not available through ordinary access. #19659 separately addresses a saved-thread feature-policy argument difference. This PR preserves local archive evidence and does not claim direct hosted verification of that primary cause. ## Model Used OpenAI GPT-6 family through Codex, with reasoning, tool use and code execution. The model is inherited; no override was used. The exact serving model ID and context-window size are not exposed by this runtime. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-06 17:52:12 -06:00
DottaandPaperclip	0819f4ea78	fix(native): verify controller ownership before retrying saved turns (#19638 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native task turns can keep a provider conversation from an earlier run. > - A copied conversation checkpoint names the new run before its controller has attached. > - A failed attachment can make recovery treat that copied identity as current execution authority. > - The current run needs its own durable controller checkpoint, or a valid staged attachment, before exact resume. > - This pull request checks that ownership and uses the normal Retry path to continue a saved user turn after verified cleanup. > - The task can continue once with its saved work and history. ## Linked Issues or Issue Description Refs #19399 and #18860. What happened? A new user comment resumed a completed native conversation. The retained runner process was gone before the new controller attached. The new run still had a copied provider session ID. Recovery selected exact resume for that new run and then failed its unchanged owner and lease checks. Ordinary first-attempt failure settlement also omitted the saved-user Retry handoff that restart recovery already uses. Expected behavior Exact resume requires current controller evidence. A saved user turn can request one normal Retry after this infrastructure failure. The normal path must verify provider stop, saved files, current user authority and all execution gates before it starts a fresh session. Steps to reproduce 1. Finish a native warm run and retain its provider checkpoint. 2. Admit a user comment as a new run and copy the provider checkpoint through the real resume producer. 3. Stop the retained runner before attachment. The real remote process check reports that the process is not running. 4. Observe that the copied provider ID selects exact current-run resume. Observe that ordinary failure settlement saves no Retry request. ## What Changed - Check a bounded encrypted controller header for fenced remote Runner recovery. Keep local Runner recovery on its existing disk-backed path. Match the run, conversation and runner identity. Preserve the existing staged-attachment path. - Share the existing saved-user Retry admission between restart recovery and ordinary terminal native settlement. Keep its exact infrastructure errors, user wake, comment, provider-turn and retry-chain checks. - Add real PostgreSQL and stopped-process regressions. Verify malformed and foreign checkpoint holds, current-owner resume, user authority and Stop. Verify saved-file recovery creates one fresh-session successor. - Document the ownership and Retry contract. ## Verification - RED: the real checkpoint rebind producer and stopped-process check selected exact current-run resume without a current controller checkpoint. - RED: ordinary settlement created zero Retry receipts for the real native failure code; expected one. - Review RED: a real current local controller file existed with no database checkpoint rows. The initial gate returned ambiguous state. The fix takes a required trusted execution-target kind and applies the database requirement only to the remote persistence path. - Initial local revision: `3662055880ccf8e09c3c50a340897e16c78627e7`, macOS ARM64, Node 26.4.0, pnpm 9.15.4, embedded PostgreSQL. - Initial server Vitest: 42 selected cases passed across controller recovery integration, remote restart adoption, ordinary run completion and retained workspace recovery. The cases cover the stopped-process producer, current/foreign/malformed ownership, automatic user continuation, Stop, unsettled turns, system wakes and a manual Retry on first use. - `pnpm --filter @paperclipai/server typecheck`: passed with this worktree's Runner, shared and SDK builds and all seven server layers. - `node scripts/check-module-boundaries.mjs`, whitespace checks and the added-code-comment audit: passed. - Review correction: `8c795e2e0e44b5209da3e228593c5cd35e948cae`. The real local/remote controller case passed against a fresh PostgreSQL cluster. It verifies local disk recovery without database rows, remote current-owner resume, and remote missing/foreign/malformed holds. Fresh `node scripts/typecheck-layers.mjs` passed all seven server layers. The existing offline chat reader now requests metadata only and retains its full independent native-state safety check. - A broader local recheck hit PostgreSQL setup/cleanup hook timeouts. These are retained as failed setup attempts; the fresh-cluster regression above passed. Required CI supplies broad final-head coverage. - Settlement-order correction: `dafb721c3bbb7ae86fbc3c9c6c953ace29ca088b`. The real PostgreSQL regression first proved both failures: Stop after an already-failed row created one Retry instead of zero; a DB-native promotion with an old legacy caller snapshot created zero instead of one. All six settlement scenarios now pass. The hook requires the local failed outcome and the canonical settled native row. Fresh seven-layer server typecheck passed again. - Required shared fixture base integrated at `d22d8f9d5ee3dc36550a94d55c1d8376f3b9aab2`. The six runner-skills fixture cases and all seven server typecheck layers passed. Required CI passed for this head: [workflow](https://github.com/paperclipai/paperclip/actions/runs/42309419592), including all server layers, tests, build, Runner verification and E2E. The final snapshot has 57 successful checks, three intended skips and one neutral result. Current-head Greptile is 5/5; both concrete review threads are resolved. The independent re-review ended without a final response; its prior Stop finding has a real RED/GREEN repair. The focused command was run from `server/`: ```sh ../node_modules/.bin/vitest run --config vitest.config.ts src/services/heartbeat/run-completion.test.ts src/services/native-runtime/native-controller-recovery.integration.test.ts src/services/native-runtime/native-session-executor.test.ts src/services/workspace-recovery.test.ts -t 'replays saved user intent\|does not assign current-run resume authority\|remote controller restart adoption\|continues a user turn once\|does not continue automatically' ``` ## Risks - A terminal failure of a fenced Runner adds one exact-run header read. The read selects one encrypted header and limits inspection to 1 MiB. Invalid or missing ownership evidence stays held. - Saved user input can request one automatic Retry. It uses normal preservation, current authorization, budget, approval, Stop and cleanup gates. Unsettled provider turns and system wakes do not start an automatic successor. - This PR changes no provider process, credential, model or lease authority checks. ## Model Used OpenAI GPT-6 family through Codex, with reasoning, tool use and code execution. The model is inherited; no override was used. The exact serving model ID and context-window size are not exposed by this runtime. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-06 17:31:23 -06:00
DottaandPaperclip	374a918d67	fix(runner): retain qualified OpenCode artifacts for cold recovery (#19622 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native recovery must keep admitted provider state and useful workspace files. > - OpenCode binds its saved session to exact executable paths and hashes. > - A replacement image can remove those files even when session data is saved. > - Retaining a manifest alone cannot restore the missing launch files. > - This pull request saves the qualified files before provider work and restores the same owned paths. > - The benefit is cold continuity without relaxing the Runner guard or replaying a prior turn. ## Linked Issues or Issue Description Refs #19558. That fix retains the selected pack for ordinary continuation. This change adds durable file retention for a cold allocation. Related #19487 adjusts the admission budget; it does not retain launch files. A search of open and closed OpenCode cold/durable artifact PRs found no duplicate. What happened? A saved OpenCode session could outlive the image that supplied its Node executable, proxy, and native executable. A new image supplied different bytes. The Runner then correctly refused the changed launch profile. The saved harness did not contain the old files. Expected behavior Core must save qualified launch files before the first turn. It must restore those exact files under the same owned account and absolute paths. Unknown ownership, missing files, and changed hashes must still stop recovery. Steps to reproduce Start an admitted remote OpenCode session. Save its workspace, encrypted checkpoint, and harness data. Destroy the original allocation and controller files. Restore into a new allocation with an image whose qualified proxy has different bytes. Resume the same session. The new opt-in local qualification command automates this path with actual OpenCode and a local synthetic model. ## What Changed - Store company-bound, encrypted launch objects and canonical run/backup references. Generate migration 0438 through the normal database workflow. - Stage qualified Node, proxy, and native OpenCode files under a private content-addressed OS-account home. Record the account UID and exact launch paths. Preserve the Rust profile guard. - Gate first provider work on durable capture under the current native claim. Bound Stop/deadline waits while the actual late transfer keeps its worker and scratch ownership until settlement. Restore cold caches with verified staging and atomic rename. Keep live reattachment and historical image-rooted sessions on their existing paths. - Reuse bounded canonical expiration owners. Atomically fence old checkpoint authority with a strict company/run-scoped retirement marker before external deletion. Keep its indexed cleanup anchor across failed deletes and restart. Preserve earlier resume handles and reject later header or source-consumer writes. Add no idle scan or network call inside a transaction. - Add storage, race, executor, filesystem, and actual Rust resume regressions. Add the opt-in local Linux qualification command and operating limits in `doc/opencode-cold-recovery.md`. ## Verification - RED before repair: real PostgreSQL/AES cold restore lacked the launch files; normal executor preparation chose the new image pack. Separate regressions exposed lost upload receipts, missing cache repair, last-owner retirement, an account-UID mismatch, and a resume-handle race during external deletion. Each expected failure was captured before its repair. - Local qualification before the final master integration: 1,169 executor tests, 233 storage/archive/admission/expiration tests, all seven server typecheck layers, migration safety, and the actual Rust cold-resume test passed. - The maintained Linux command passed with two cached immutable Paperclip images whose proxy hashes differ. It removed the original account/controller trees, restored the production published cwd under a new lease, resumed the same actual OpenCode session, kept a useful file and prior model context, and made one new synthetic model request. No external model call or network was available in either container. - The object-store fixture uses real encrypted StorageService streams with an independent owned local directory. It is not hosted S3 acceptance or a complete task HTTP/actor workflow. Historical tasks still require normal effects reconciliation and recovery after deployment. - Review regressions reproduced partial-cache publication, resume-versus-deletion, fresh-writer and source-consumer resurrection, and unbounded Stop/deadline waits before each repair. They now pass through real filesystem, encrypted storage, PostgreSQL, and SDK boundaries. File validation is split from database ownership (384/831 source lines). - Clean runtime head `aadab7acc69ff959678030db46ffe2dcfbefe28f`: the maintained actual OpenCode cold qualification passed with `dirty:false`. It preserved the provider session and recorded launch paths, restored useful workspace content and prior model context, and made exactly one resumed synthetic model request. The 314,560,000-byte launch capture took 23.118 seconds; the later harness was 833,536 bytes. Subsequent repairs change backup-reference retirement and governed continuity-cache classification; capture, restore, launch, and published-workspace inputs are unchanged, so that runtime receipt remains scoped to those paths. The continuity-break consumer is verified separately. - Retention repair `c14d386cecf7123257918ba6a8d47307df2da397`: 68/68 affected real PostgreSQL/AES storage tests and 15/15 focused executor authority/admission cases pass. All seven server typecheck layers pass (source layers and unchanged test shards, with the final 590-file test layer repeated after fixture edits). All review threads have evidence-backed resolutions. Its subsequent CI caught the separate lifecycle-classification gap described below. The refreshed Paperclip review reported paused work, not a completed new review. - Final head `4d1cd3e3309e328b11abc85adc6d78c6d0c7f55d`: all 19 focused root-entry/executor cases and all three source declaration layers pass. [Greptile reviewed this exact head at 5/5](https://github.com/paperclipai/paperclip/pull/19622#issuecomment-6695984874). Final-head [CI verification](https://github.com/paperclipai/paperclip/actions/runs/42308908167/job/128214100339) passes: 60 successful checks, five conditional skips, and the explicitly neutral paused Paperclip review. No check is pending or failing. - CI also caught the launch cache missing from the session-root lifecycle contract. Both the writer scan and a real governed-replacement consumer failed locally before repair: old launch bytes remained in the active root. The cache now archives with the prior session state; the consumer verifies exact bytes are preserved in the archive and absent from the active root. - The latest review found that backup retirement could delete an independent run pin while a selected session retained the checkpoint. A real PostgreSQL/AES regression failed before the repair. Backup-only ownership now uses a NULL-safe exact backup predicate; the run pin belongs to fenced checkpoint expiration. The regression verifies checkpoint decryption and launch-file restoration after pending-backup cleanup, followed by deletion only after removing the final resume handle. - An executor first-session case checks new-session durable capture through the real capture helper and receipt propagation into the actual harness manifest at the backup boundary. Real storage cases verify the run-profile receipt and canonical backup/run pins. - Sequential closure reuse and simultaneous first admission are different scopes. A deterministic two-uploader test measures both independent uploads, one verified ready winner, loser-only deletion, and usable restoration for both runs. Each capture is bounded to 512 MiB and five minutes; up to three pending receipts per run are retained. There is no claim of global single-flight capture or a company-wide egress bound. - Earlier optional warm-ownership verification encountered missing local binaries. After exact-source setup, all nine ownership assertions passed; two other suites encountered local PostgreSQL setup/teardown contention, so those interrupted attempts are not passes. Required current-head CI supplies the broad final coverage. No deployment or hosted historical recovery is claimed. ## Risks - This adds two tables and a forward migration. The file object is bounded to 512 MiB. Capture adds one initial transfer; later harness archives exclude those large files. Company-scoped ready objects are shared and retained until the last canonical owner retires. Concurrent initial admissions can duplicate bounded transfers; each keeps independent ownership and cancellation rather than waiting on another controller. A continuity break retains the controller cache with its prior state until normal session-root cleanup, adding up to one bounded launch closure per retired state. - Cold continuity requires the same qualified Linux platform, supported runtime dependencies, owned account UID/home, and published absolute cwd. A mismatch fails closed. No alias or Rust guard change is included. - Existing image-rooted profiles have no retained file object. This change cannot backfill missing historical binaries. A fresh governed session must reconcile prior effects first. - Live reattachment does not read artifact objects. Storage or provenance failures before a new turn hold admission. They do not grant new execution, model, tool, or replay authority. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, repository tools, code execution, and local browser tools. The exact served model ID and context window are not exposed in this session. No claim is made for unavailable model metadata. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-06 17:30:00 -06:00
DottaandClaude Opus 5.5	15ee6a25a6	fix(native): re-bootstrap a claimed Daytona run killed before runner launch (#19616 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs on Daytona start a `paperclip-runnerd` process in a sandbox. Restart recovery decides what to do with each live run after the control plane restarts. > - #19583 lets recovery start the same-run bootstrap again when the Postgres controller checkpoint proves that the runner received only bootstrap commands. > - A live replay found an earlier window. The control plane died after the executor claimed the run and staged the sandbox workspace, but before it asked for a runner process. No runner checkpoint existed, so #19583 did not apply. > - Recovery classified that run as `ambiguous_remote_recovery_evidence` and blocked the task. The task also showed "Workspace repair required", which did not describe the real cause. > - This pull request proves the "nothing reached a runner" case from durable evidence and starts the bootstrap again. It also names the restart in the settled hold note. > - The benefit is that a restart at any point of Daytona startup before runner launch continues automatically. ## Linked Issues or Issue Description What happened? A chaos replay killed the control plane about 2 s after a claimed ACPX Claude run started on Daytona. The kill came after `stage.workspace` and environment acquisition, before `native.process_start_requested`. The restarted server took over because the previous controller process was dead. It then returned `ambiguous_remote_recovery_evidence` with disposition `blocked`. The task rested blocked as "Workspace repair required". Evidence in the saved database at that point: coordinator `attempt: 1`, `controllerGeneration: 1`, previous controller identity recorded and dead. Native workspace sync was `prepared`. There were no runner checkpoint rows, no session checkpoint, no process PID or start time, no provider session, and no execution-start or runner-sourced event. The local session root was an empty directory. Expected behavior When durable evidence proves that no runner process was requested and no controller checkpoint exists, recovery continues the run with a fresh bootstrap. Any evidence that a runner may have started keeps the hold. Steps to reproduce 1. Start a native run on a Daytona environment. 2. Kill the control plane after workspace staging and before the executor requests the runner process. 3. Restart the control plane. 4. See that restart recovery blocks the task with `ambiguous_remote_recovery_evidence`. Root cause: `server/src/services/native-runtime/native-restart-recovery.ts`, function `classifyNativeRunnerRecoveryEvidence`, remote branch. The only no-checkpoint remote rule required an unclaimed coordinator (`attempt: 0`, no controller identity). A claimed run with no launch evidence fell through to "ambiguous". Refs #19583, #16349 ## What Changed - `native-restart-recovery.ts`: add the `remote_runner_never_launched` proof, which leads to `bootstrap_incomplete`. It applies only to a classic sandbox run with a prepared workspace and the preparation fencing version. The run must have a real executor claim (controller identity and lease owner) whose controller is gone (takeover allowed). It needs no durable checkpoint, no session checkpoint, no process fields, no provider session, and no provider evidence. The run must also have no `native.process_start_requested`, process identity, `adapter.invoke`, `tool.execution.started`, or runner-sourced event. The same three-attempt budget applies. The shared event query moved into `hasNativeExecutionStartEvidence`. - `native-session-executor.ts`: `restoreRemoteBootstrapCheckpointForRecovery` removes an empty session root that preparation created. Without that step, the fresh bootstrap read it as `durable_identity_unreadable`. - `execution-recovery-resolution.ts`: the new `executionRecoverySettlementNote` names a settled `native_restart_recovery_blocked` hold as a stop after a server restart. It no longer shows "Workspace repair required" because a later workspace export was retried. Other causes keep their notes. - `doc/run-log-events.md`: document the new reason. ## Verification - Red, then green: `cd server && npx vitest run src/services/native-runtime/native-runner-restart-recovery.integration.test.ts -t "unlaunched\|never requested\|Postgres checkpoint"`. Before the fix, the proven case (lease live with a dead owner, and lease expired) and the budget case returned `blocked` / `ambiguous_remote_recovery_evidence`. The empty-root executor case failed. - Fail-closed cases: these keep the hold: `native.process_start_requested`, a process identity event, `adapter.invoke`, a runner-sourced event, a recorded PID, a recorded provider session, and a spent budget. A live claiming controller returns `awaiting_evidence` / `controller_still_alive`. The existing "contradictory evidence" cases (`claimed` with no controller identity, `launch`, `process`, and others) stay blocked. - Wording: `server/src/services/execution-recovery-resolution.test.ts`. - Regression set (10 files, including `workspace-recovery`, `native-safe-replacement`, `heartbeat/recovery`, `native-session-executor`, and `native-restart-recovery`): 2035 passed, 14 skipped. - Live Daytona replay of the failing chaos task, 3 runs, sequential, ACPX Claude, double SIGKILL, current master plus this change. Image: content-matched `paperclip-daytona-runner@sha256:5a29a797…` built from `c6eecac1e0`, with runner inputs pinned to it, because the image-only build sat in the Actions queue. Result: 3/3 PASS (R1 pass; R2, R4, R5, R6 and T pass). Each run's faults landed after the runner launched. Recovery waited on the live remote controller lease, then continued without a Retry. Run 3 continued through `reattach_remote_runner`. The saved logs for runs 1 and 2 do not name their claim kind. The earlier pre-launch window (the subject of this PR) is not hit reliably by replay timing. The integration tests cover that window with the same evidence shape as the saved failing database. ## Risks - The proof depends on the executor writing `native.process_start_requested` before a backend may spawn a runner (`onSessionAdmission` in `executeNativeSession`). A future change that spawns a runner earlier would weaken this proof. The fail-closed tests pin the event as the gate. - A fresh bootstrap acquires a new sandbox. The staged sandbox from the dead attempt keeps its lease until run settlement releases all run leases. It never ran a runner. - The note change affects only settled holds with the cause `native_restart_recovery_blocked`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, GitHub CLI) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 17:17:14 -06:00
DottaandClaude Opus 5.5	7941f6b051	fix(runner): show OpenRouter's invalid model ID as a model rejection (#19548 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs use the Paperclip runner with drivers such as OpenCode > - When a provider rejects the selected model, the product shows "Model unavailable" and offers a model choice > - OpenCode reports provider HTTP errors as `APIError` with the message under `data`, not at the top level > - The runner then reported "Provider turn failed", so an invalid model ended as a generic adapter failure > - This pull request classifies OpenRouter's invalid model ID with a closed provider-failure category, and makes the server keep that classification > - The benefit is that a person sees "Model unavailable" and the real provider message, and can choose another model ## Linked Issues or Issue Description What happened? A chat evaluation selected a model ID that OpenRouter does not know. OpenCode reported `turn.failed` with `error: { name: "APIError", data: { message: "<model> is not a valid model ID", statusCode: 400, isRetryable: false } }`. The run was saved with `errorCode: adapter_failed`. The task chat showed "The OpenCode run failed before it completed." and no "Model unavailable" marker. The provider notice said only "OpenCode session failed.". Expected behavior A non-retryable rejection of the selected model is saved as `native_provider_model_rejected`, so the product shows "Model unavailable" and the model choice. The notice shows the provider's message. Steps to reproduce 1. Configure an OpenCode agent with an OpenRouter model ID that does not exist. 2. Send the agent a message. 3. Before this change, the run fails as `adapter_failed` and the notice says "OpenCode session failed.". ## What Changed - `drivers/opencode/provider-failure.ts`: a non-retryable OpenCode provider error whose message says the model is not valid (`not a valid model`, `model_not_found`, model not found or not supported) gets the closed category `model_unavailable`. A provider that allows a retry keeps the existing category. - `native-session-runtime.ts`: a non-recoverable failure with `model_unavailable` is a model rejection. The runtime reads the closed category, not provider text. - `native-session-executor.ts`: a non-recoverable `NativeProviderTerminalFailure` that the runner marked as a model rejection is saved as `native_provider_model_rejected`. Before, every such terminal was saved as `native_provider_terminal_failed` before the marker was read. - `drivers/opencode/opencode-server-driver.ts`: the provider notice summary shows the provider message from `data` when there is no top-level message. Provider text is not used as the turn failure message, so it does not reach `run.error` or Sentry. - Tests: provider-failure mapping (retryable and non-retryable), runtime classification by category, driver notice summary with no failure message, and an executor test for the saved code and terminal disposition. ## Verification - `npx vitest run src/native-session-runtime.test.ts src/drivers/provider-failure.test.ts src/drivers/opencode` in `packages/paperclip-runner`: 16 files, 412 tests passed. - `npx vitest run src/services/native-runtime/native-session-executor.test.ts` in `server`: 1,152 tests passed. - Negative controls: reverting the provider-failure mapping, the runtime check, or the executor branch each makes its new test fail. ## Risks - Low risk. Classification uses a closed category and a runner-set marker, not provider text. - The model check on the provider message is text-based inside the OpenCode mapper. A provider that uses "not a valid model" for another reason with a non-retryable error will show "Model unavailable". - Provider text appears in the notice summary, as provider notices already did for top-level messages. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 17:08:04 -06:00
Devin Foley Claude Opus 5.5 Paperclip Paperclip	ee6c101854	Reject failover backups behind the accepted event cursor (#19613 ) A verified failover backup can still trail events already accepted by the controller. Compare the backup produced cursor with the registered controller state, or the supported local journal before registration, and reject stale files before remote sync. Keep the permanent replay-conflict classification ahead of generic Runner errors. Validation: 1,192 native session and execution-driver tests and 10 workflow tests pass locally, together with the maintained low-memory Runner test project typecheck and fresh server declarations. All applicable checks on `cd8c9cd07c` are green (56 success, three skipped); exact-head Apex review is 5/5 with no unresolved findings. One e2e shard was retried after GitHub reported lost self-hosted runner communication, and the same-head rerun passed. No acknowledgement was lowered, provider replay authorized, or recovery files discarded. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-Authored-By: Paperclip <Paperclip-Paperclip@users.noreply.github.com> Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-11-06 13:55:43 -08:00
Nicky LeachandClaude Opus 5.5	a7d3d4b78c	fix(runner): rotate local and external authority through one resumable path (#19547 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Paperclip Runner moves a native session from one run to the next with an authority rotation, for Codex and for every ACPX provider > - A rotation archives the prior run's controller state under `authority-epochs/` and archives the Runner state, locally or on an external Runner host > - If the run fails after the rotation but before the new Runner starts, retry preparation restores the prior run's controller checkpoint into the session root > - The retry then finds an active `control-plane` next to the archive marker and fails with `native_runner_authority_archive_conflict`. Every later run for the session fails the same way > - The local and external rotations were two copies of the same steps, with different retry rules, and the sandbox archive script had a third set of rules > - This pull request gives every provider and Runner placement one resumable rotation path, with the same retry rules for controller state and Runner state > - The benefit is that one failed startup no longer makes a native session unusable ## Linked Issues or Issue Description No public issue exists. Bug description: What happened? A warm Codex session on an external (plugin) Runner failed its first attempt after the authority rotation finished. The two automatic retries both failed with `native_session_recovery_failed: Error: native_runner_authority_archive_conflict`, and the run stopped. On disk, the session root had `authority-epochs/epoch-<key>/control-plane/control-plane-state.json` and an active `control-plane/control-plane-state.json` with the same identity (in this case, also the same sha256). The remote Launcher had already archived the Runner state under the same key. Expected behavior A retry resumes the interrupted rotation. The rotation already supports this when only the archive marker exists. Steps to reproduce 1. Run a native Codex or ACPX session on an external Runner. 2. Start a new run for that session. Make the new Runner fail to start after the rotation finishes. 3. Let the run retry. Retry preparation (`restoreWarmRunnerdController` / `restorePerTurnRunnerdController` → `restoreNativeControllerProjections`) restores the prior run's controller checkpoint into the session root. 4. The retry fails with `native_runner_authority_archive_conflict`. All later runs for the session also fail. The new `authority rotation after the prior controller state is restored` tests reproduce steps 2–4 directly. Paperclip version or commit `master` at `35280516c0`. Deployment mode Hosted, with a remote (plugin) Runner environment. Agent adapter(s) involved Codex (native Paperclip Runner). ACPX providers use the same rotation code. ## What Changed - One rotation path. `rotateAuthorityEpoch` replaces `rotateLocalAuthorityEpoch` and `rotateExternalAuthorityEpoch`. The only difference between Runner placements is a `RunnerStateArchive` (`read`, `archive`): - `localRunnerStateArchive(root)` for Runner state under `<root>/runner` - `#externalRunnerStateArchive()` for a plugin task or a sandbox host - One caller. The transport chooses the rotation source (the live control plane, or the latest archive), then makes one call. Locally owned state is still quarantined on failure. - Committed rotation, normalized once. Retry preparation restores the prior run's controller checkpoint, but an earlier attempt may already have archived that authority and its Runner state. `takeCommittedRotation` runs right after the transport reads the control plane. When the epoch archive for the active copy's exact identity exists, it removes the restored copy and continues on the archive-only path. Every later check (the warm-handoff evidence read, the source-attachment settlement read, and the rotation) then sees the state the rotation committed, instead of reading retired live Runner state. A copy that still holds an unfinished warm transition, staged attachment, or pending `run.attach` fails closed. - Restored controller state. When the epoch archive exists, an active control plane with the same schema and the exact archived identity is a copy of authority that the rotation already retired. The rotation removes it and resumes from the archived snapshot. This compares identity, not bytes, because a checkpoint restore serializes the state again and drops fields such as `ackedSourceSeq`. Any other active state still fails closed. - Duplicate Runner state. Local Runner state now follows the computer Launcher's rules: move it, accept it already archived, drop a byte-identical live copy beside the archive, or fail closed. The local archive also checks that the live state is suspended before it moves it. - Sandbox archive script. The script is now `REMOTE_RUNNER_AUTHORITY_ARCHIVE_SCRIPT` and follows the same rules. It uses `cmp` for the duplicate check and fails closed when `cmp` is not available. - Sandbox host preparation after a resumed rotation. A resumed rotation archives the remote Runner state without preparing the host first. When `archiveExternalRunnerState` succeeds, the server now records the archived snapshot path. The launch's host preparation (`inspectRemoteHarnessState`) validates that snapshot instead of the missing live `runner-state.json`, with the same provider-state and identity checks. Only these host preparation checks use the archived snapshot; close checkpointing and backup-restore verification always inspect the live `runner-state.json`. The SSH staging check treats it as present, so retired state is not staged back. - Host preparation. The external archive prepares its host only when it reads live Runner state. Before, only one of the two caller branches prepared the host. A resumed rotation finds the state already archived, and preparing again could stage it a second time. ## Verification - `npx vitest run src/live/runnerd-codex-transport.test.ts -t "rotation"` in `packages/paperclip-runner`: 16 passed. - External and local rotations with a byte-identical, a re-serialized, and an other-authority restored control plane. The first two resume from the archived snapshot. The third fails closed and keeps the copy. - Local rotation with an identical duplicate Runner state (resumes) and a different duplicate (fails closed and keeps it). - The existing crash-before and crash-after remote archive retry cases. - Before the source change, the byte-identical external case failed with `native_runner_authority_archive_conflict`. - `npx vitest run src/live/runnerd-codex-transport.test.ts -t "transport retry after a committed rotation"` in `packages/paperclip-runner`: 4 passed. These go through the transport's real start path. A completed warm handoff and a source attachment hook now resume from the archive; before the normalization, both read the archived live Runner state and failed. An unfinished warm transition and a pending attachment fail closed; before, the pending attachment was silently dropped. - `npx vitest run src/services/native-runtime/native-session-executor.test.ts -t "remote authority archiv"` in `server`: 6 passed. Five new cases run the real archive script with `sh` against temporary files: a new archive, an existing archive, an identical duplicate, a different duplicate, and no state. The existing mocked archive test also passes. - `npx vitest run src/services/native-runtime/native-session-executor.test.ts -t "resumed rotation archived"` in `server`: 2 passed. A resumed sandbox with a retained backup goes through the real archive script and host preparation. It failed with `runner_harness_state_mismatch: resumed_unavailable` before the host preparation fix. A settled close checkpoint with no live Runner state then fails closed with `runner_remote_checkpoint_incomplete`, rather than checking the archived snapshot. Without a completed rotation, a missing live state still fails closed. - `pnpm run typecheck:typescript` in `packages/paperclip-runner` and `node scripts/typecheck-layers.mjs` in `server`: pass. - The full `runnerd-codex-transport*.test.ts`, `cursor-cloud-runner-launch.test.ts`, and adoption test files fail on the same 110 tests with and without this change. These tests need the built `paperclip-runnerd` binary, which needs `cargo`. `cargo` is not installed locally, so CI covers them. ## Risks - The change is only on the rotation path. A rotation that has no archive marker behaves as before: it reads and checks live Runner state, then archives the control plane. - A rotation removes an active control plane only when an archive for that exact identity exists and the active copy has the same schema and identity. A new run's authority has a different identity, so it cannot match. - On a resume, the rotation now uses the archived controller snapshot, not the caller's copy. The archive is the exact pre-rotation state. A checkpoint restore can omit fields from it. - An external Runner whose options define `readRunnerState` and `archiveExternalRunnerState` but not `prepareExternalRunnerState` can now rotate a live control plane. Before, that branch required the prepare hook. The plugin and sandbox drivers define all three. - The sandbox duplicate check needs `cmp`. A host without `cmp` fails closed as before. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, test runs). It diagnosed the failure from the run events, the server logs, and the on-disk controller and Launcher state, then wrote the fix and the tests. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 13:27:55 -08:00
DottaandClaude Opus 5.5	da9a4cb96c	fix(native): retire a finished local Runner that stopped before suspending (#19597 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The native Runner path runs each per-turn agent turn through a runnerd process, and the next turn must prove that the previous Runner is finished before it takes over the saved session root > - Under load, a local turn can have its result accepted while its `runner.suspend` command does not finish, so the Runner's own receipt stays unsuspended > - The retirement proof that handles the same case for sandbox Runners refused every local Runner with `target_not_sandbox_runner` > - So the next turn failed with `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate` and its session root was quarantined > - This pull request extends the retirement to a local Runner when the prior Runner process is proven stopped by exact PID and start time > - The benefit is that a follow-up turn (for example, after a user answers a question) continues instead of failing ## Linked Issues or Issue Description Refs #19525 (keeps the accepted result in the same incident; this is the next run). What happened? In the runner E2E cell `local-session-integrity.runner-acpx-claude.local.structured-question-restart-resume`, the first turn asked a question and its result was accepted. The Runner then logged `runner did not prove durable suspension before checkpoint` (`runner.suspend` pending, Runner lifecycle `ready`). After a server restart and the answer, the next run failed with `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate; retirement_refused=target_not_sandbox_runner`, and its root was quarantined. Expected behavior When the prior local Runner is provably stopped and every other retirement proof holds, the next run retires that owner as a continuity break and continues. If the stop cannot be proven, the run keeps the named rejection. Steps to reproduce 1. Run a local per-turn native turn whose `runner.suspend` does not finish after its result is accepted (host load average ~40). 2. Start the next turn for the same session. 3. See `retirement_refused=target_not_sandbox_runner`. ## What Changed - `server/src/services/native-runtime/native-restart-recovery.ts`: add `proveNativeLocalRunnerStopped`. It uses the existing restart-recovery checks: the recorded Runner PID is dead or now belongs to a process with a different start time; its recorded process group is not alive; each recorded provider process is dead or recycled. A missing PID, start time or process group, a live recorded process, or a start time that cannot be compared is not proof. When settlement already cleared the run's process fields, the saved `native.local_process_stopped` receipt is the Runner proof. Extract `nativeProviderProcessIdentities` (checkpoint plus run events) so restart recovery and this proof read provider PIDs the same way. - `server/src/services/native-runtime/native-session-executor.ts` (`retireFinishedPerTurnRunnerdOwner`, `readSettledRunnerdProjection`): a local target is now eligible. For a local Runner, the projection accepts the Runner's own unsuspended receipt for the same identity and unfinished Runner lifecycle commands (`runner.suspend`, `runner.drain`, `runner.shutdown`) only. All other proofs are unchanged: committed result, released lease, no recovery state, stopped controller, delivered terminal, other commands complete, tool inputs answered, quarantine copies. The process proof runs before retirement and again just before the archive. - `doc/execution-semantics.md`: describe the local case. - Tests: seven integration cases in `native-recovered-owner-followup.test.ts` and ten unit cases in `native-restart-recovery.test.ts`. ## Verification - Red before the fix: the four new integration cases failed with `retirement_refused=target_not_sandbox_runner`, which is the incident error. - Green after the fix: `pnpm exec vitest run src/services/native-runtime/native-recovered-owner-followup.test.ts` 35/35. The new cases prove that a stopped Runner is retired (root archived under `continuity-breaks`, no quarantine), and that a live recorded Runner (`runner_process_alive`), a missing identity (`runner_process_identity_missing`), and a pending non-lifecycle command (`journal_commands_unsettled(turn.start.pending)`) keep the rejection. All existing sandbox cases still pass, including "keeps an unsuspended Runner receipt decisive" for sandbox targets. - `native-restart-recovery.test.ts` unit cases: recycled PID, live process, unknown start time, live group, live provider, missing PID or start time. - `native-session-executor.test.ts` + `native-restart-recovery.test.ts`: 1190 pass. `native-runner-restart-recovery.integration.test.ts` + `native-restart-recovery-isolation.test.ts` (shared provider-PID extraction): 81 pass. `tsc --noEmit -p server/tsconfig.json`: pass. - Live: `pnpm test:e2e:runner -- --max-parallel 1 --max-automatic-retries 0 --id local-session-integrity.runner-acpx-claude.local.structured-question-restart-resume`, 3 runs at `eec57dd5ed`: 3/3 PASS. The host load average was ~15 (the incident had ~40), and `runner.suspend` finished in all three runs, so these runs show no regression but did not exercise the new retirement path. The integration cases cover that path. ## Risks - The new path lets a follow-up archive a local root whose Runner receipt is unsuspended. This is safe only if the Runner is stopped, so the proof uses exact PID and start-time identity and fails closed on any uncertainty. `ps` start times have one-second precision; a sub-second difference is treated as the same process (not proof). - A zombie-only process group on macOS can report as alive. That keeps the rejection, as before. - No quarantine behavior changes. A refused retirement still quarantines the root as before. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, file edits, GitHub CLI). Extended reasoning on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 15:11:00 -06:00
DottaandClaude Opus 5.5	586caa5f49	fix(native): hand a shutdown-interrupted startup to the next controller (#19596 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The native Runner path runs an agent turn through a runnerd process that the control plane owns > - A restart of the control plane must keep the task going without a click, if that is safe > - A graceful shutdown closes the HTTP listener first and then drains the heartbeat runs > - A native run that was between provider dispatch and runnerd backend setup in that gap failed with `local_runtime_api_listener_unavailable`, so the drain did not see it as running and did not suspend it for the next boot > - This pull request makes that failure a restart detach: the run stays running with its claim, the drain suspends it, and the next controller adopts it > - The benefit is that a SIGTERM in this window no longer spends a failure and a same-run retry, and the task continues by itself ## Linked Issues or Issue Description Refs #19530 (the same restart window, before provider dispatch). What happened? In a restart chaos replay (ACPX Claude, local runner), a restart-recovered run got SIGTERM while `createRunnerdBackendWithinSessionClaim` built the runnerd backend. `resolveLocalRuntimeApiUrl()` threw `local_runtime_api_listener_unavailable` because shutdown had already closed the listener. The run settled `failed`. The drain then logged `restartSuspendedRunIds: []`. The next boot used the one bootstrap retry. A second SIGTERM hit the same window, and the task stopped at a manual Retry. Expected behavior A run that this process cannot continue because it is shutting down is handed to the next controller. It is not settled as failed. Steps to reproduce 1. Start a native Runner task and restart the server (K1) so that startup recovery resumes the run. 2. Send SIGTERM while the resumed run sets up its runnerd backend. 3. See the run fail with `local_runtime_api_listener_unavailable` and the drain report no suspended runs. ## What Changed - `server/src/runtime-api.ts`: `resolveLocalRuntimeApiUrl()` throws the new `LocalRuntimeApiListenerClosedError` when this process had a listener and closed it (the `listening` flag is false or the `close` event fired). The error message stays `local_runtime_api_listener_unavailable`, so the existing classifiers do not change. A listener that never started still throws the plain error. - `server/src/services/native-runtime/native-session-executor.ts`: for a local Runner, the startup failure handler treats `LocalRuntimeApiListenerClosedError` like a restart detach. It stops lease renewal, keeps the claim and throws `NativeControllerDetachedForRestartError`. Heartbeat already keeps such a run `running`. A remote Runner keeps the same-run bootstrap retry from #19462, because startup recovery has no process evidence to adopt a claimed remote startup (`native-runner-restart-preparation.integration.test.ts` still covers that path). - `doc/execution-semantics.md`: describe the new rule. - Tests: a heartbeat regression that closes the real runtime API listener before backend setup, then drains and restarts; a unit test for the closed versus never-started listener; two `vi.mock` factories keep the real module exports. ## Verification - Red before the fix: `pnpm exec vitest run src/__tests__/heartbeat-native-restart-suspension.test.ts -t listener` failed with `restartSuspendedRunIds: []` and the run `failed` (`local_runtime_api_listener_unavailable`, `bootstrap_retry`). - Green after the fix: the same test passes. The drain suspends the run, the run stays `running` with `awaiting_runner_reattach`, and the next controller claims it and reaches the backend once. - `pnpm exec vitest run src/__tests__/heartbeat-native-restart-suspension.test.ts src/http/server.test.ts src/__tests__/server-startup-feedback-export.test.ts src/services/native-runtime/native-session-executor.test.ts src/services/native-runtime/native-recovered-owner-followup.test.ts`: 1333 tests pass. One timing test in `native-session-executor.test.ts` failed once under host load average ~30 and passed alone. - `pnpm exec tsc --noEmit -p server/tsconfig.json`: pass. - Live, restart chaos harness, local ACPX Claude (`claude-sonnet-5`), `--rc06-masked`, this branch at `a35ec8b845` (before the remote-scope commit `5dc63f24fb`, which does not change the local path). The original replay schedule no longer reached the window, because the task now ends before the second fault. So I moved the faults earlier: K1 at 10 s, then K2 as soon as the recovered server is ready. - K1 + K2, 2 runs: both runs PASS with R1 pass. Each drain logged `restartSuspendedRunIds: [<run>]` (before: `[]`). The next boot classified the run `resume_dead_runner` and the same run completed. No `local_runtime_api_listener_unavailable` failure. - K1 + K2 + K2, 2 runs: each drain suspended the run. The fourth boot blocked the run with `native_restart_recovery_blocked`, because three restart claims used the three-attempt recovery budget. Retry then completed the task (R3 pass). That budget count is a separate open item. This pull request does not change it. ## Risks - Low. The new path applies only after this process closed its own listener, which happens only during shutdown. The run then waits for the drain or the next boot, as for the other restart detaches. - If shutdown does not reach the drain (for example SIGKILL after the listener closes), the run keeps its live claim until the claim expires. Then startup recovery handles it as a hard restart. This is the same as any run that is running at a hard kill. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, file edits, GitHub CLI). Extended reasoning on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 15:10:44 -06:00
DottaandClaude Opus 5.5	d728be8fbf	fix(native): re-bootstrap a Daytona runner restarted before session.open (#19583 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs on Daytona start a `paperclip-runnerd` process in a sandbox. The local control plane keeps the runner's controller state in an encrypted Postgres checkpoint. > - When the control plane restarts, restart recovery classifies each live native run and decides to reattach, resume, bootstrap again, or hold the task. > - A restart chaos wave killed the control plane twice while a Daytona runner was connecting. The run was claimed and the sandbox runner was launched, but `session.open` never completed. No provider evidence existed. > - Recovery waited correctly for the old 60 s controller lease to expire. Then the classifier had no rule for a claimed remote bootstrap. It returned `ambiguous_remote_recovery_evidence` and blocked the task as "Workspace repair required". > - The controller writes each command to the Postgres checkpoint before it sends it. After the lease expires, that checkpoint is the complete record of what reached the sandbox runner. > - This pull request lets recovery prove from that checkpoint that no provider work exists, and then start the same-run bootstrap again. > - The benefit is that a restart during Daytona runner startup continues automatically instead of stopping the task. ## Linked Issues or Issue Description What happened? A restart during Daytona remote runner startup (native ACPX Claude, two SIGKILLs about 20 s apart) left the task blocked. The run history showed `remote_controller_lease_live` twice, then `ambiguous_remote_recovery_evidence` with disposition `blocked`. The task showed "Workspace repair required" (`native_restart_recovery_blocked`) and did not continue. Expected behavior When the durable controller record proves that the runner received only bootstrap commands, recovery continues the run with a fresh bootstrap. Any evidence of provider work keeps the hold. Steps to reproduce 1. Start a native run on a Daytona environment. 2. Kill the control plane after the sandbox runner connects and before `session.open` completes. 3. Restart the control plane and wait for the old controller lease to expire. 4. See that restart recovery blocks the task with `ambiguous_remote_recovery_evidence`. Root cause: `server/src/services/native-runtime/native-restart-recovery.ts` `classifyNativeRunnerRecoveryEvidence`, remote branch. It admitted only an unclaimed bootstrap (`remote_execution_never_claimed`, #16349) or a run with a provider checkpoint. A claimed bootstrap with no provider evidence fell through to `ambiguous_remote_recovery_evidence`. The decrypted checkpoint of the failed run held `connectionCount: 1`, `run.prepare` delivered once and pending, `session.open` pending and never delivered, no committed events, and `ackedSourceSeq: 0`. Refs #16349, #19524 ## What Changed - `native-bootstrap-proof.ts`: move the incomplete-bootstrap predicates from the executor into a small pure module. Restart recovery and the executor now use the same proof. No behavior change. - `native-runner-checkpoint.ts`: add `nativeRunnerCheckpointProvesIncompleteBootstrap`. It reads only a checkpoint with the bootstrap parts (header, cursor, `command:1`, `command:2`), checks the run, runner instance, and session identity, and applies the shared proof. A remote runner journal is not on this host, so the proof uses the controller record alone. - `native-restart-recovery.ts`: for a classic sandbox run whose owner lease has expired, with a durable checkpoint, no session checkpoint, no provider evidence, and no staged attachment, claim `bootstrap_incomplete` with the reason `remote_bootstrap_proven_incomplete` when the checkpoint proof holds. At three provider attempts, block with `execution_recovery_budget_exhausted`. - `native-session-executor.ts`: add `restoreRemoteBootstrapCheckpointForRecovery`. For a `bootstrap_incomplete` recovery on a sandbox target, it restores the Postgres checkpoint over the local external-state marker. The existing incomplete-bootstrap proof then checks the restored state before it quarantines the root and bootstraps again. - `doc/run-log-events.md`: document the new disposition reason. ## Verification - Red, then green: `cd server && npx vitest run src/services/native-runtime/native-runner-restart-recovery.integration.test.ts -t "remote bootstrap"`. Before the fix, the proven case and the budget case returned `blocked` / `ambiguous_remote_recovery_evidence`. - New cases: the proven claimed bootstrap gets `bootstrap_incomplete` (attempt kept, generation +1, run still `running`, no recovery action). The hold stays for a queued `turn.start`, an acknowledged event, adopted state, no checkpoint, and a spent budget. The executor helper restores the checkpoint only for a bootstrap recovery on a classic sandbox and only over a marker for the same run. - Regression set: `native-runner-restart-recovery.integration`, `native-session-executor`, `native-restart-recovery`, `native-bootstrap-retry.integration`, `heartbeat/recovery`: 1383 passed, 14 skipped. - Live Daytona replay of the failing chaos task (double SIGKILL during remote runner startup), 3 runs: REPLAY_RESULTS ## Risks - A fresh bootstrap acquires a new sandbox. The old sandbox keeps its lease until run settlement releases all run leases. The old runner gets no turn command, because only the controller issues one and the new controller never connects to it. - If `session.open` reached the old runner, the old provider session can exist without a turn. No provider work or tool call can happen in it. - Any checkpoint shape other than the bootstrap-only shape keeps the previous fail-closed hold. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, GitHub CLI) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 14:43:22 -06:00
DottaandPaperclip	b55d8a9201	fix(runner): retain admitted OpenCode pack on ordinary continuation (#19558 ) ## Thinking Path > - Paperclip manages AI agents and preserves their work across runs. > - Native OpenCode turns can restore a saved provider session after the prior daemon stops. > - The Rust provider binds that saved state to exact qualified launch paths and artifact bytes. > - Ordinary continuation selected the current provider pack, although live reattachment retained the admitted pack. > - A new proxy or a diagnostics setting could therefore replace the launch profile behind saved state. > - This pull request retains the recorded pack for ordinary OpenCode continuation with a verified durable binding. > - The benefit is compatible continuation without weakening artifact integrity or granting replay authority. ## Linked Issues or Issue Description Refs: #17512, #18237. These added saved authority and compatible pack reuse. This change covers ordinary OpenCode continuation. What happened? An ordinary remote OpenCode continuation could select a newer pack while restoring its existing provider session. The Rust recovery guard then rejected the changed launch profile. The same error can occur after a controller release changes the bundled proxy. Expected behavior A continuation with a valid saved binding and pack receipt uses the same admitted artifacts. If those artifacts are unavailable, it holds the saved state instead of selecting another pack or creating a provider session. Steps to reproduce 1. Admit a remote OpenCode session with a qualified protected provider pack. 2. Persist its pack receipt and suspended durable Runner state. 3. Change the controller bundle to a valid pack with a different proxy digest. 4. Start an ordinary continuation with diagnostics enabled. The new executor regression fails before the repair because selection rejects the available original pack in favor of the new bundle. It passes after the repair. ## What Changed - Retain the saved provider pack for ordinary OpenCode continuation with a verified durable session binding. - Keep provider-pack retention independent of surviving-daemon identity: a new daemon records its current binary, and an authorized continuity break restores normal compatible pack selection. - Test retained pack selection after a proxy upgrade and fail closed when only a newer pack is available. - Test the real Rust executor with changed, valid proxy file bytes. It rejects drift without changing saved state and accepts the original artifacts again. - Document the selection rule and its limits. ## Verification - Initial prevention focused Vitest: 220 passed across native session execution, pack authority, pack verification and pack preparation. The executor regression was observed failing before the production change. - Rust native provider tests: all five OpenCode tests passed. These include the existing drift guard, actual changed proxy bytes, original artifact restoration and restart-lost turn behavior. - Advisory review follow-up: reproduced stale daemon identity in the actual executor and incompatible pack selection after a governed continuity archive. Both fail before their corresponding repairs and pass after them. Added an executor regression for a missing authority receipt; it remains at the current release integrity gate without creating a new receipt. - Updated head `639c080856` integrates master `17901d74ba` by normal merge. Affected Vitest: 202 passed across four files, including all new review regressions; all seven server typecheck layers passed. Exact-head CI and fresh Greptile/Paperclip review are pending. The prior head had 57 successful checks plus successful Snyk and Greptile 5/5; those results do not qualify this revision. - Environment: macOS arm64, Node 26.4.0, locked pnpm 9.15.4 dependencies, controlled remote command boundary and real Rust provider executor. No paid provider calls or hosted retries. - This is a bounded prevention change. It does not deliver historical provider artifacts to a replacement sandbox or replacement controller. Missing saved receipts or unavailable old artifacts remain a separate cold-recovery gap. No full hosted recovery is claimed. ## Risks - A continuation now requires its exact recorded pack when it has a durable binding. A newer installed pack cannot substitute for missing historical bytes. - Fresh launches, other providers and explicit continuity replacement keep their existing paths. Current ownership, company, model, permission and receipt binding checks remain in effect. - The repair does not relax the Rust launch-profile guard, replay prior tool effects, or authorize a new provider conversation. Existing published review receipts remain history. ## Model Used OpenAI Codex, GPT-6 family. The host does not expose the exact model ID or context window. Capabilities used: reasoning, code editing, shell execution, tests and authenticated read-only browser diagnostics. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-06 14:21:17 -06:00
Nicky LeachandPaperclip	1391f64551	Add gated release/resume methods to plugin Runner owners (#19584 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Core owns Runner tasks through a lease-bound plugin owner. > - The SDK already defines release and resume in #19468. > - The owner still expects an accepted result for both operations, so it cannot use those SDK results. > - This pull request adds both methods behind the existing capability, warm-task and account checks. > - It uses the SDK parser to verify the result and task ID, and preserves the owner's cancellation checks. > - This prepares the owner boundary for later lifecycle integration. It does not start automatic release or resume. ## Linked Issues or Issue Description Problem or motivation The plugin Runner owner cannot forward the existing SDK release/resume pair. Its local result mapping rejects their valid results. Later lifecycle code needs both operations on the same checked task owner. Proposed solution Pass the existing description capability to the owner. Add explicit release and resume methods and use the SDK's operation-result parser. Require warm-task support and a selected account. Reject calls and late receipts after owner cancellation or detachment. Alternatives considered Adding automatic release here would also require connection quieting, settled-event drain, pending-ticket retry ownership and safe disposal. Keep those changes together in the next integration. Do not add a new flag, schema, timer or lifecycle mode here. Related: #19468. I checked ROADMAP.md and found no duplicate planned item. An open-PR title search for release/resume found no overlapping pull request. ## What Changed - Add the capability-gated owner facade for Codex and ACPX. Resume returns only `resumed` or `unavailable`. - Use the SDK parser for exact task-ID and operation-result validation. - Pass `releaseResumeVersion` from the checked description and test that it does not cause an automatic operation. - Cover account-preserving owner rebind, missing prerequisites, malformed results, ambiguous errors, cancellation and late replies. - Document the caller's remaining settlement, connection, ticket and attachment duties. ## Verification Passed locally on Node 26.12.0 with the pinned workspace dependencies: - In `server/`: `pnpm exec vitest run src/services/native-runtime/plugin-runner-task.test.ts` — 74 tests passed. - In `server/`: `pnpm exec vitest run src/services/native-runtime/native-session-executor.test.ts -t 'uses a task-only'` — 3 cases passed; 1,160 unrelated cases were not selected. - Built the real workspace SDK dependencies and Runner TypeScript. A focused strict TypeScript program for the owner and its tests passed. - `git diff --check` and the added-line secret/PII scan passed. The first root test command selected a nonexistent Vitest project name and ran no tests. Running the test from the server package fixed the command. The first focused TypeScript setup could not find Node types; pointing it to the package's installed type definitions fixed that setup. The full regression suite and release checks run in GitHub. ## Risks The facade has no automatic caller. Existing warm timeout, bootstrap, outbound connection and pool behavior stay in place. No plugin capability is advertised or task control channel enabled by this pull request. Future integration must serialize release with new work and disposal. It must quiet the owned connection, keep the exact pending ticket after an ambiguous reply, authenticate the source again and finish ordinary run attachment before new work. This change does not qualify provider parking, cold recovery, retention policy or deployment. ## Model Used OpenAI GPT-6.1 Sol, exact model ID `gpt-6.1-sol`, High reasoning. Used code execution and targeted tests. The context-window size was not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-06 11:42:32 -08:00
DottaandClaude Opus 5.5	07ddf9a5c2	fix(native): keep accepted results when runner close cannot prove suspension (#19525 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runs persist an accepted result first. The Runner then closes and suspends its session. > - Under heavy host load, the Runner close can fail to prove a durable suspension after the result is accepted. > - The executor then kept its finalization lease. The heartbeat finalizer got `native_finalization_lease_busy`, and the run ended `failed/adapter_failed`. > - The provider work was complete. Only Runner bookkeeping failed, so the run must not fail. > - This pull request hands the exact lease to finalization and completes the run from its accepted result. > - The benefit is that an accepted result stays accepted, independent of host load. ## Linked Issues or Issue Description What happened? A native ACPX Claude run asked a structured question. The control plane accepted its result (`Waiting for…`) and recorded `run.terminal` with `runTerminalState=succeeded`. The Runner close then took about 8.6 s. Its `runner.suspend` command ended `failed`, but the durable Runner state was `suspended` with a matching identity. The close threw `NativeSessionCloseUnrecoverableError`. The executor emitted `native.post_result_failure` (`category: runner_settlement_incomplete`, `resultAlreadyAccepted: true`) and rethrew `native_session_lease_lost`. Its finalization lease stayed held (its TTL is 20 minutes). `finalizeNativeRun` got `native_finalization_lease_busy`, and the run ended `failed/adapter_failed`. No restart was in progress. Host load average was about 40. Expected behavior After the control plane accepts a result, finalization bookkeeping must not fail the run. The run settles from its accepted result. Repair of the Runner checkpoint stays with the existing next-run admission rules. Steps to reproduce Run `pnpm exec vitest run src/services/heartbeat/runtime-execution.test.ts -t "accepted local result"` in `server/`. The test uses the real executor, the real control-plane result commit, and embedded PostgreSQL. The provider boundary accepts a result and then throws `NativeSessionCloseUnrecoverableError`. On the base revision, the finalization row keeps the executor lease (`phase: workspace_finalizing`) and the finalizer reports `lease_busy`. Paperclip version or commit Base `ff1d1dea5d`. The incident was on `819420a60a`. ## What Changed - `native-session-executor.ts`: after the rejected failure write proves an accepted result under the exact, unexpired lease, the executor releases that lease to finalization. It uses the same exact-authority predicates as the normal release. Before, the lease stayed held until its TTL expired. - `runtime-execution.ts`: for `runner_settlement_incomplete` on a local workspace (no workspace sync and no remote workspace authority), Paperclip records `workspace_finalize` as succeeded, finalizes the accepted result, and returns it to the ordinary completion path. - New `native-accepted-result-settlement.ts`: rebuilds the adapter result from the durable accepted result row, its terminal, the run's native session, and the accepted usage checkpoint. The result carries the closed `nativePostResultFailure` diagnostic. It contains no provider text. - Other post-result categories (for example `io_not_found`) keep their workspace-failure handling. They now also reach finalization at once, instead of after the lease TTL. - `doc/execution-semantics.md`: describes the settlement. ## Verification - Red, then green: `settles an accepted local result when runner close cannot prove suspension` and `keeps workspace failure handling for other accepted-result failures on a local workspace` in `server/src/services/heartbeat/runtime-execution.test.ts`. Both failed on the base revision with the executor lease still held. Both pass with this change. - Safety case `settles an accepted local result ... (workspace-check-failed)`: when the `workspace_finalize` check fails, the run keeps the existing failure path. Case `terminal-unreadable`: when the accepted result cannot be rebuilt, only a failed `workspace_finalize` record is written. - `pnpm exec vitest run src/services/heartbeat/runtime-execution.test.ts`: all pass (some cases time out at 30 s only under host load above 40; they pass on a narrow rerun). This includes the 14 existing post-result diagnostic and authority cases (expired, replaced owner, generation, attempt, boot, pid, process start, stopped, lock busy, and others). - `pnpm --filter @paperclipai/server typecheck`: pass. - Live cell `local-session-integrity.runner-acpx-claude.local.structured-question-restart-resume` (real ACPX Claude, local runner): 3 runs at host load average about 40. Runs 1 and 2 passed. Run 3 hit the fault naturally: the first run emitted `native.post_result_failure` (`runner_settlement_incomplete`) and now ended `run succeeded` (before this change it ended `failed/adapter_failed`). The cell then failed later, in the resume run after the answer: `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate; retirement_refused=target_not_sandbox_runner`. That is a separate follow-up (see Risks). ## Risks - A changed lease handoff. The executor releases the lease only after the rejected failure write proves an accepted result under the exact owner, attempt, controller generation, controller identity, and an unexpired lease. A stale or replaced owner does not release anything. A failed release is bounded and best effort. The old behavior then applies. - The run now completes as `succeeded` while the local Runner checkpoint is not proven suspended. The next run on the same session then meets the existing fence: the continuity-break retirement (`retireFinishedPerTurnRunnerdOwner`) accepts only sandbox Runners, so a local Runner is refused with `target_not_sandbox_runner` and its state root is quarantined. This PR does not change that fence. Extending the retirement to local Runners needs its own proof that the local Runner process stopped, and is a follow-up. - Settlement happens only when the accepted result and its terminal are rebuilt first, and the `workspace_finalize` record then succeeds. A failed branch or plugin workspace check keeps the failure path. - Synced and remote workspaces are not changed. They keep their copy-back retry path. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, test runs). Extended reasoning was on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Related: #19126 added the closed post-result diagnostic that this change uses. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 12:36:34 -06:00
DottaandClaude Opus 5.5	beaa9a622d	fix(native): re-bootstrap a runner killed before session.open (#19524 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs start a local `paperclip-runnerd` process. The control plane keeps a durable state root for each session. > - When the control plane restarts, restart recovery classifies each live native run. A run whose runner died before any provider identity existed gets the `bootstrap_incomplete` disposition. Recovery then quarantines the old state root and starts the bootstrap again. > - The executor accepts that disposition only when the durable state root proves that no provider work started. The old proof required `connectionCount === 0` and every command `pending`. > - A restart chaos wave killed the server about 2 s after runnerd started. The runner had connected and completed `run.prepare`, but `session.open` was still pending. No event was committed, so no provider session or turn existed. > - The proof rejected that root. The run failed with `runner_state_identity_mismatch: bootstrap_not_proven_incomplete`, and the task stopped with `native_continuation_requires_reconciliation`. Nothing continued automatically. > - This pull request widens the proof to the exact window where provider work cannot have started, and keeps every other shape fail-closed. > - The benefit is that a restart during native runner bootstrap continues automatically for every native provider. ## Linked Issues or Issue Description What happened? A control plane restart that lands after the native runner connected and acknowledged `run.prepare`, but before `session.open` completed, fails the run with `adapter_failed` and `runner_state_identity_mismatch: bootstrap_not_proven_incomplete`. Startup recovery chose `bootstrap_incomplete` (controller generations 2 and 3), but the executor proof in `migrateRunnerdStateRootForExecution` refused it. The durable control-plane state had `connectionCount: 1`, `run.prepare` completed, `session.open` pending, no committed events, and `ackedSourceSeq: 0`. The runner journal had `nextSourceSeq: 1`, an empty outbox, and the provider state was `prepared` with no provider session ID. Steps to reproduce 1. Start a native task (seen with OpenCode on a local runner; the window applies to every native provider). 2. Kill the control plane about 2 s after runnerd starts, after `run.prepare` completes and before `session.open` completes. 3. Let the control plane restart. Expected behavior Recovery quarantines the state root and starts the bootstrap again. The task continues without a user action. Actual behavior The run fails and the task is blocked with `native_continuation_requires_reconciliation`. ## What Changed - `runnerdStateIsIncompleteBootstrap` (native-session-executor.ts) now also accepts a connected root when all of these are true: no committed event, `ackedSourceSeq === 0`, only `run.prepare` and `session.open` commands, `session.open` not completed, `run.prepare` pending or completed, no warm transition or staged attachment, and the bootstrap was not adopted. - When `runner/runner-state.json` exists, the runner journal must agree: empty outbox, `nextSourceSeq === 1`, `ackedSourceSeq === 0`, and no completed `session.open` or other processed command. An unreadable runner journal stays fail-closed. This covers events that the runner emitted but the controller never committed. - The remote "claim an untouched session" path keeps the old never-connected proof (`runnerdStateIsUnconnectedBootstrap`), so its behavior does not change. - Tests: a red test with the production state shape (connected, `run.prepare` completed, `session.open` pending), plus fail-closed counter-tests for a completed `session.open`, committed or acknowledged events, a turn command, a failed `run.prepare`, an adopted bootstrap, a warm transition, an uncommitted runner event, a runner-completed `session.open`, a runner-processed turn, and a torn runner journal. ## Verification - Red before the fix: `pnpm exec vitest run src/services/native-runtime/native-session-executor.test.ts -t "native incomplete-bootstrap evidence"` failed on "allows a fresh bootstrap after connect and run.prepare when session.open never completed" (`expected false to be true`). Green after the fix: 13/13. - The proof run against the kept durable state root from the failing wave (read only) returns `true` after the fix. - Related suites: `native-session-executor.test.ts` and `native-restart-recovery.test.ts` filtered to `bootstrap\|restart\|remote\|unstarted\|untouched`: 301 passed. - `tsc --noEmit -p server/tsconfig.json` passes. - Live restart chaos proof: see the comment on this PR. ## Risks - The proof now accepts a root where the runner received `session.open` but did not complete it. The runner can have started a provider process or created an empty provider session in that window. Recovery only proceeds when restart classification already proved that the runner process and its group are dead and no known provider process is alive. No provider turn can exist, because the controller never issued a turn command. The worst case is an orphaned empty provider session, not repeated work. - Every state with committed or acknowledged events, a completed `session.open`, a turn, a warm transition, an adopted bootstrap, or an unreadable runner journal still fails closed. - Out of scope: when the proof fails legitimately, the task offers the existing execution-blocker Retry. In the failing wave the issue interruption also showed a budget wait (`userAction: budget`). That is the RC-06 accounting issue and belongs to a separate fix. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude Code with tool use (shell, file edits, test execution). Default context window. Extended reasoning was available. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 12:30:50 -06:00
DottaandClaude Opus 5.5	e680fa6814	fix(native): retry startup interrupted by a closed runtime API listener (#19462 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs prepare a remote workspace, then start a runner and a provider session > - A graceful server shutdown closes the HTTP listener before it drains runs > - A run that is still preparing then asks for the local runtime API origin, and the lookup fails with `local_runtime_api_listener_unavailable` > - Recovery did not know this code. It treated the failure as an ambiguous provider state and blocked the task behind a board-owned action, although no runner or provider process existed > - This pull request classifies the closed listener as a definite pre-session failure > - The benefit is that a deploy no longer strands a preparing task. The same run gets a bounded bootstrap retry that the next server process claims ## Linked Issues or Issue Description What happened A native Codex task on a remote sandbox spent about ten minutes in workspace preparation. A deploy started a graceful shutdown during that time. When preparation finished, the executor called `resolveLocalRuntimeApiUrl()` to build the tool authority. The listener was already closed, so it threw `local_runtime_api_listener_unavailable`. No runner backend or provider session had started. The failure classifier returned the generic `native_session_interrupted` code. `nativeProviderRecoveryEvidence` then returned `ambiguous_state`. The run became a terminal failure, the task became blocked, and the recovery action told the board to resolve an ambiguous session and to not open a replacement session. The same signature occurred on three other runs during an earlier deploy. Expected behavior A failure that occurs before any runner or provider process exists is safe to retry on the same run. The task must stay in progress with a scheduled, bounded bootstrap retry. If a provider event or checkpoint exists, recovery must stay operator-owned. Steps to reproduce Start a native run. Close the server's HTTP listener before the executor creates the runner backend. Observe a `terminal_failure` with `recoveryMode: "ambiguous_state"` and a blocked task. The new integration test reproduces this with the real `resolveLocalRuntimeApiUrl()` and a real Postgres database. Environment Server native runtime, remote sandbox target, `codex_app_server` driver. Related recovery work: #19249, #19304, #19305. ## What Changed - `nativeSessionFailureSourceCode` returns a new `local_runtime_api_listener_unavailable` source code for this error. - `nativeProviderRecoveryEvidence` treats that code as definitely pre-session. The existing guard stays: the run must have no checkpoint and no durable provider event. - The chat failed-run retry accepts the code in its list of ordinary pre-session failures, so an exhausted startup retry can still be retried from chat. - New integration tests in `native-startup-retry.integration.test.ts` cover the retry path and the operator-owned path when a provider event exists. ## Verification - Red before the fix: the new test failed with `phase: "terminal_failure"`, `recoveryMode: "ambiguous_state"`, `recoverable: false` and a board-owned recovery. This matches the failed run on the deployed instance. - Green after the fix: `cd server && npx vitest run src/services/native-runtime/native-startup-retry.integration.test.ts` passes 26 of 26 tests. The retry test asserts `retryable_failure`, `bootstrap_retry`, backoff timing, an agent-owned `resume_native_run` wake, and a claim through `claimNativeSessionResumptions`. - Safety case: when a durable `turn.started` event exists, the same error stays `terminal_failure` with `ambiguous_state` and a blocked task. - `npx vitest run src/services/native-runtime/native-session-executor.test.ts -t "listener\|classif\|pre-session"` passes. - Not run: a live deploy that interrupts a preparing sandbox run. The test uses the real listener lookup and the real recovery tables, but not a real sandbox. ## Risks - Low risk. The change only affects runs that fail with this exact error and have no checkpoint and no provider event. - The retry uses the existing bootstrap budget of three attempts with backoff. Repeated failures still stop and hand the task to the board. - The GitHub callback bridge that started in the sandbox before the failure is a per-attempt helper. The existing cleanup path releases it. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through Claude Code with tool use, code execution and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 08:27:59 -06:00
DottaandClaude Opus 5.5	6ca7b151f7	feat: compact an agent's context from the task Properties panel (#19378 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A task's Properties panel shows the assignee agent's context window (paperclipai/paperclip#19273) > - The board can see that the context is nearly full, but it cannot act on it > - A full context makes the agent compact at an unplanned time, often in the middle of work > - This pull request adds "Compact now" to the Context hover card > - The benefit is that the board can compact the agent's context between runs, at a time it chooses ## Linked Issues or Issue Description Refs paperclipai/paperclip#19273 Problem The Context row shows how full the agent's context window is. The board cannot ask the agent to compact it. Proposed solution A "Compact now" button in the Context hover card. It runs the harness's own compaction: Codex `thread/compact/start`, and Claude's `/compact` command through ACP. Alternatives considered A separate PRP `session.compact` command. Both harnesses run compaction as a turn, so a compaction turn keeps the existing durable turn binding, interruption, and recovery. A separate command needs a second turn lifecycle. ## What Changed - UI: the Context hover card shows Compact now with three states: available, Compacting…, and unavailable with a reason (agent mid-run, harness unsupported, task closed). The Storybook story shows each state. - Server: `POST /api/issues/:id/context/compact` (board only, `decideIssueAccess` `issue:mutate`, activity log `issue.context_compaction_requested`). It queues a short run marked by `contextSnapshot.contextCompactionRequestId`. `GET /issues/:id/context-usage` returns the `compaction` state. - Server: a compaction run does no task work. The native status arbiter preserves the task status. The lifecycle skips handoff, liveness continuation, review-path disposition and the issue comment, as it does for goal-control runs. - Runner TS: `startTurn({ compact: true })`. The Codex harness session sends `thread/compact/start`. The runnerd transport maps it to PRP `turn.start` with `compact: true` (Codex) or `text: "/compact"` (Claude/ACPX). The native runtime option `sessionCompaction` requires a resumed session and settles with a synthesized yielded result. - Runner Rust: `turn.start` accepts `compact: true` for Codex. `CodexProvider::start_compaction_turn` sends `thread/compact/start` and binds the turn from the `turn/started` that follows the empty response. - Fix (from #19273): Codex emitted `context.compacted` on both the start and the end of one compaction item. It now emits it once, and the count is distinct by `compactionId`. - Fix (from #19273): the conversation event view dropped `context.compacted`, so a finished run never showed the divider. The view now keeps compactions, plus context readings for runs that compacted. A finished run's compactions show as thread dividers. - Fix: the Codex harness stamps before/after sizes on the compaction event. Codex reports the compacted size before the compaction item ends, so the old neighbour-based guess showed the wrong size. ## Verification - `pnpm -r typecheck`, `pnpm build`, `pnpm check:token-gates`, `node scripts/storybook-catalog.mjs`: pass. - New tests: `server/src/__tests__/context-compaction.test.ts`, `status-arbiter.test.ts`, `issue-context-usage.test.ts` (duplicate compaction counts once), `ui/.../context-usage-row.test.tsx`, `native-session-runtime.test.ts` (context compaction), Codex driver and runnerd transport tests, Rust `codex_compaction_turn_binds_the_turn_reported_after_its_empty_response`. Rust `codex_provider` suite: 115 passed. - `pnpm test:run` locally: one failure in `chat-channels.integration.test.ts`. It reads `.claude/skills/slack/SKILL.md` from the main checkout, which is not present in this worktree. It is not related to this change. - Live, local worktree instance, Codex agent on the Paperclip runner: clicked Compact now. The run succeeded. Context went from 35k to 9.4k tokens, and later 9.4k to 9.2k. The task stayed In Review. The thread shows "Context compacted · 9.4k → 9.2k tokens" and the meter dropped. - Live Claude proof: pending. The local Claude agent needs an Anthropic API key. claude-agent-acp handles `/compact` as an echo-less turn, and the runner already maps its "Compacting completed." notice to `context.compacted`. ## Risks - A compaction run resumes the agent's session. If the session cannot resume, the run fails with `native_context_compaction_unavailable` and the task is unchanged. - Compact now is not available while the agent runs on the task, because compaction interrupts the active turn. - The conversation view now returns context readings for runs that compacted. Runs that never compacted return the same events as before. - An old runner binary ignores `compact` and rejects the `turn.start` because it has no text. It does not start an ordinary turn. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 22:07:12 -06:00
DottaandClaude Opus 5.5	36643d9378	fix(native): retire finished owners whose continued turn rotated authority (#19373 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The native runner runs per-turn agent sessions in managed sandboxes. A controller keeps a durable session root for each session. > - When a per-turn owner finishes without a saved runner checkpoint, the next turn retires that owner. Retirement first checks that the session root holds only known entries. > - The runner transport writes two entries that the check does not know: `authority-epochs/` and `runtime-context.json`. A continued turn writes `authority-epochs/` when it rotates authority from the turn before it. > - So the third turn of an ephemeral-lease task cannot retire the second turn's owner. The run fails with `native_continuation_requires_reconciliation` and the task waits for a user message. > - This pull request classifies every session-root entry in one contract module, and adds a test that scans the writers so the list cannot drift again. > - The benefit is that follow-up turns continue without manual help, and any future refusal names the unexpected entry. ## Linked Issues or Issue Description Refs #18846, Refs #19061, Refs #19355 What happened? On staging, the third turn of every task for one agent (gpt-5.6-sol, native runner, ephemeral sandbox lease) failed. The error was `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate; retirement_refused=unexpected_root_entry`. The first two turns succeeded. The task then went to `blocked` with "Saved work recovered. Send a message to continue." Expected behavior The third turn retires the finished second-turn owner and continues the conversation. Steps to reproduce 1. Assign a task to a native agent that uses an ephemeral sandbox lease. 2. Let the first turn finish. Reopen the task with a comment so a second turn runs and finishes. 3. Reopen the task again. The third turn fails with the error above. Paperclip version or commit Staging near 9a79384958; reproduced locally on master `7c0922a7b6`. ## What Changed - Add `server/src/services/native-runtime/native-session-root-entries.ts`. It is the single contract for top-level session-root entries: - Continuity state (moves with a continuity break): the existing entries, plus `authority-epochs` and `launch-preparation.json`. - Launch entries (stay in place, carry no authority): `runtime-context.json`, `diagnostics`. - Held entries (retirement still refuses): `plugin-runner-state.json`, `cleanup-activation.json`, `cursor-cloud`. - `readSettledRunnerdProjection` uses this contract. The refusal now names the entries. A held entry reports as `held_root_entry(cursor-cloud)`. An unclassified entry reports as `unexpected_root_entry(unrecognized-state)`. Names are bounded to three, limited to safe characters, and have identifiers replaced with `<id>`. - Add `native-session-root-entries.test.ts`. It scans every local root writer (the executor, the Cursor Cloud runtime, the provider-pack authority, the controller migration, the runner transport, and the launch-preparation store). It reads string and template literals, same-file constants, `posix.join` and `path.join`, and the harness persistence directory names. It fails when a writer adds an unclassified entry, or reaches the root through a new unresolved variable. - Add three cases to `native-recovered-owner-followup.test.ts`: a finished owner whose root has `runtime-context.json`, an archived authority epoch, `diagnostics`, and `launch-preparation.json` is retired, and the epoch and launch preparation move into the archive; an unknown entry still holds the owner and is named; a held entry is named as held. ## Verification - Red before the fix (same string as staging): `retires a finished owner whose continued turn left runtime context and an archived authority epoch` received `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate; retirement_refused=unexpected_root_entry`. - Green after the fix: - `cd server && npx vitest run src/services/native-runtime/native-recovered-owner-followup.test.ts src/services/native-runtime/native-session-root-entries.test.ts` (34 passed) - `cd server && npx vitest run src/__tests__/native-harness-backup-expiration.test.ts src/services/native-runtime/native-session-resume.test.ts src/services/native-runtime/native-session-executor.test.ts` (1313 passed) - Live confirmation on staging is pending. After deploy, run a three-turn task on an ephemeral-lease codex agent. The third turn must succeed. Its log must show "finished without a saved runner checkpoint; starting a new runner". ## Risks - Retirement now accepts two more entries. `authority-epochs` only holds epochs that the transport archived after it verified a suspended runner, so it carries no live owner claim. `runtime-context.json` and `diagnostics` are launch inputs and logs. Unknown entries still refuse retirement. - A continuity break now also moves `authority-epochs` and `launch-preparation.json` into the archive. This is the intended owner: these files belong to the retired authority lineage. If they stayed, a later cold rotation could read an archived epoch of the retired lineage. - I could not read the staging sandbox. The evidence for the exact entry is from code: the second turn's transport writes `authority-epochs/` during authority rotation, and every launch writes `runtime-context.json`. The refusal now names the entry, so any other entry will be visible in the next failure. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edit, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 19:37:01 -06:00

1 2 3 4 5 ...