PaperClipAI

0 Fork 0

mirror of https://github.com/paperclipai/paperclip.git synced 2026-11-12 22:56:58 +01:00

Author	SHA1	Message	Date
DottaandClaude Opus 5.5	c966334ec3	fix(recovery): settle a run-less lease once its stamped holder is proven gone (#20026 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The control plane recovers in-progress records after a server restart through a shared ownership path > - Run-less sandbox leases (login, probe, harness verification) were released only by a five-minute idle sweep > - That sweep releases by age: it leaves a dead holder's sandbox for minutes and takes a live login session's sandbox > - This pull request settles a run-less lease only when its stamped holder is proven gone > - The benefit is no stuck sandbox after a restart, and no live session loses its sandbox ## Linked Issues or Issue Description Related: #19965 (named the owner of login/probe leases), #19964 (the run-lease settle this mirrors). No duplicate PR was found. What happened? The final Daytona chaos wave `w-20261108T083036-s9401` recorded I2 `environment_login_probe_lease` (`pid_dead`) on 2 records. These were the codex agent's harness-verification leases. A restart killed their holder. The leases stayed `active` until the five-minute idle sweep released them. The same sweep releases any run-less lease after five idle minutes, also when its holder is alive. Expected behavior A run-less lease is released as soon as its stamped holder is proven gone. A lease whose holder is alive is never released, however long it is idle. Steps to reproduce 1. Start a Daytona wave: `node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env daytona --daytona-image <image>` (K1/K4 faults). 2. Let a fault kill the server while an agent's harness verification holds its sandbox lease. 3. See the reconciler report I2 `environment_login_probe_lease` until the idle sweep releases it. Paperclip version or commit `master` at `8475826d2d`. ## What Changed - New `settleEnvironmentLoginProbeLease`, registered on `environment_login_probe_lease`. It runs only for a stamped holder that is proven gone, and compares and swaps on that exact stamp. - It moves the lease to `pending_cleanup` with the recovered-orphan reason, so the cleanup sweep destroys the sandbox in its next tick. - It leaves the lease alone when a legacy shutdown fence applies, or when another live lease holds the same provider resource. - The idle sweep for run-less leases now covers only legacy rows saved before holders were stamped (no `leaseController` key). Stamped rows are never released by age. - `holderUnchanged` is now exported from the run-lease settle so both settles share it. - The controller stamp (`leaseController`, and every other `ControllerProcess` stamp) now records the platform deployment when the platform names one (Railway: project, environment, service, deployment, volume). `controllerStamp` passes it to `proveOwnerGone`. A container redeploy changes the PID space, and the old stamp would read `different_process_space` (unprovable) forever. Now the existing `deployment_replaced` proof shows the old holder gone. This also covers `environment_run_lease_active` and the other kinds built on `controllerStamp`. - `currentDeploymentIdentity` in `owner-proof.ts` now delegates to `currentControllerDeployment`, so there is one parser. - Data doc: added the final Daytona wave, the six Daytona-only crash points (4 pass, 2 not reached, no violations), and the cause of the roughly 450 s codex TTFT. Runs waited behind the agent's harness verification, which restarted after each fault; boot itself took about 32 s. - Restart-chaos README: corrected which Daytona crash points a plain task reaches. ## Verification - Red, before the change: `environment-login-probe-lease-ownership.test.ts` failed 3 of 6. No settle happened, and the idle sweep released 3 live holders' leases. - Green: `pnpm exec vitest run` passes for `environment-login-probe-lease-ownership`, `environment-run-lease-ownership` and `heartbeat-orphaned-active-lease-sweep` (45/45), and for `environment-lease-owner-stamps`, `heartbeat-observed-workspace-preservation`, `ownership-reconciliation-report`, `heartbeat/recovery` and `workspace-recovery-warm-handoff`. - Redeploy: a stamp from a replaced Railway deployment settles with `deployment_replaced`. A stamp from the same deployment in another PID space stays `active`. This was red when `controllerStamp` dropped the deployment. - Live-owner counter-cases: the current process, another live PID, another PID space, and a lease re-stamped after the proof all stay `active`. A read-only pass never settles. - Not re-run live: the Daytona wave and crash points ran before this fix. The next Daytona wave should show no `environment_login_probe_lease` I2. ## Risks - A stamped run-less lease whose live holder never releases it, because of a bug in that process, is no longer released after five idle minutes. It is released once that process exits. This is the intended trade: never take a live holder's sandbox. - On an ephemeral server without a platform deployment identity (not Railway), a stamp from a replaced container is still unprovable. Such a run-less lease stays `active` until the provider expires the sandbox, and the reconciler reports it as unprovable. Before this PR, the five-minute sweep released it by age. This is the same open gap as unstamped legacy leases on an ephemeral server: there is no operator action yet. It is listed as a Phase 2 follow-up. - Legacy unstamped run-less leases keep the age fallback. These are rows written before #19965, and no current code writes them. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-08 06:43:45 -06:00
DottaandClaude Opus 5.5	20cc6faca3	feat(recovery): settle legacy shutdown fences of proven-gone servers on the shared path (#20017 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a server shuts down gracefully during a sandbox run, it commits a fence on the run's lease so no cleanup worker destroys the sandbox before its files are exported > - The fence names its server only by boot id. It is released when its owner settles, or when its 30 s deadline (renewed every 10 s) passes. After a restart, the sweep still waits for that deadline even though the old server is gone > - The durable ownership plan proves owners by identity through one shared prover and settles them from one reconciler > - This pull request stamps the committing process on the fence and settles it when the shared prover shows that process gone. The outcome is the same safe one an expired deadline gives: the source is preserved and the restore failure recorded > - The benefit is that the fence releases on proof, not age, and a fence whose owner may still be alive (another host, or an older marker) keeps its deadline ## Linked Issues or Issue Description Refs #19831 (plan, Phase 2 sections A and C), #19838 (inventory: "Legacy graceful-shutdown workspace settlement fence", and "A boot-id mismatch alone does not release"). What happened? `legacy_shutdown_settlement_fence` was report-only, and its owner stamp had only a boot id. A fence left by a crashed or replaced server could be released only by its deadline. Expected behavior A fence whose committing process is proven gone (a dead or reused PID in the same PID space) settles from the recovery sweep and from the shared reconciler. An unprovable owner keeps the renewed deadline as its fenced takeover. ## What Changed - `legacy-shutdown-workspace-settlement.ts`: - `beginLegacyShutdownWorkspaceSettlement` stamps `controller` (`currentControllerProcessStamp`, no I/O). - New `legacyShutdownSettlementOwnerStamp`. - `allowLegacyShutdownWorkspaceCleanup` takes `ownerProvenGone`, which replaces only the deadline check, and only while the marker still names that process. Every other guard is unchanged: owner stop, competing allocations, in-flight cleanup. - The sweep also proves the owners of fences from other boots that are still before their deadline. - New `settleLegacyShutdownSettlementFence` for the reconciler. - `recovery/ownership-registry.ts`: the kind uses the full stamp when present and registers the settle. - The settling-kinds list in `ownership-reconciliation-report.test.ts` is updated. ## Verification - `heartbeat/recovery.test.ts`: 100 pass. Three new tests fail on `master` and pass here: - the fence stamps the committing process; - a previous server proven gone settles before the deadline (restore failure recorded); - the reconciler settles only the proven-gone owner (a read-only pass changes nothing). Another host and a boot-id-only marker stay pending until their deadline, then expire as before. - `ownership-reconciliation-report.test.ts` (25) and `environment-retained-orphan-sandbox.test.ts` (19) pass. - The server package typecheck passes. ## Risks - A proven-gone owner now expires the fence up to 30 s earlier than before. Expiry preserves the source (stop-and-retain or a `retain_on_failure` hold) and never destroys provider files. - Markers written before this change have no `controller` and behave exactly as before. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-08 06:15:36 -06:00
DottaandClaude Opus 5.5	82a1fda9b1	feat(recovery): settle observed-workspace preservation fences on the shared path (#20022 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a native run on a remote sandbox fails mid-turn with unsaved files, Paperclip pins an observed-workspace preservation fence. The sandbox is stopped and retained, never deleted, and the run is settled only after the provider confirms the stop > - The fence is held by the run's expired coordinator lease. Its release proof is the provider's stop receipt, and only the environment-settlement worker acted on it > - The ownership reconciler could not see these fences at all: they sit on the preserved-workspace recovery action, which is resolved (no replay) while the fence still waits > - The durable ownership plan has every in-progress record visible to the one reconciler, and settled from it once its owner is proven gone > - This pull request makes the fence visible, proves it by the provider stop, and settles it through the same function the worker already uses > - The benefit is that a fence whose sandbox stopped can no longer sit unseen, and expiry alone never releases one ## Linked Issues or Issue Description Refs #19831 (plan, Phase 2 sections A and C), #19838 (inventory: "Observed-run workspace preservation fence"). What happened? `recoveryActionKind` lists only unresolved recovery actions, so `observed_workspace_preservation_fence` reported nothing: its fence lives on a resolved action. The kind registered no settle, and its fence owner never carried the provider proof. Expected behavior Every `stop_pending` fence is listed. The fence is `gone` (`provider_stopped`) once its run's one remote lease has a termination receipt and a confirmed stop-and-retain. The reconciler then settles it with `settleStoppedObservedWorkspace`, which re-checks every fence field under its own locks. An expired coordinator lease without that receipt is `unprovable` and is never released. ## What Changed - `recovery/ownership-registry.ts`: - `recoveryActionKind` takes `includeResolved`, which only the observed fence uses. - `resolveFenceHolders` adds `provider: "stopped"` for observed fences whose run lease carries the stop proof. - The kind registers a settle that calls `settleStoppedObservedWorkspace`. A source-busy lock conflict is kept for a later pass. - `native-observed-workspace-preservation.ts`: pinning a fence wakes the ownership worker in the same transaction. - The settling-kinds list in `ownership-reconciliation-report.test.ts` is updated. ## Verification - `heartbeat-observed-workspace-preservation.test.ts`: 29 pass. The new test "settles the fence from the shared reconciler once the provider stop is proven, never before" fails on `master` and passes here: - a pinned fence with an expired lease is `fence_expired_without_provider_proof` and is not settled; - after the provider stop, a read-only pass sees `provider_stopped` and changes nothing; - the settle moves the fence to `stopped` and the coordinator to `terminal_failure`, and the saved files are untouched. All 15 existing "keeps unsafe observed source held" cases and the stale-evidence cases still pass. - `ownership-reconciliation-report.test.ts` (25) passes. - The server package typecheck passes. ## Risks - The kind's listing now includes resolved recovery actions that carry a `stop_pending` fence. Like every recovery-action kind, it is an evidence-predicate scan; no new index is added. - The settle is the existing worker's function with all its fence checks. The reconciler is a backstop for that worker, not a new release rule. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-08 05:56:08 -06:00
DottaandClaude Opus 5.5	484170d13b	feat(recovery): prove copyback owners on the shared path by their PID space (#20018 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a server restarts in the middle of copying a native run's workspace back, the run keeps a receipt that names the controller doing the copyback, so a second controller never writes the same workspace > - A successor may take over only after proving that controller and its copyback children stopped. Three places did this with their own hostname check and then judged the PID on this host: the finalization takeover, the local stop proof, and the board confirmation > - A hostname is not a PID space: two containers can share one, so a dead PID here says nothing about a controller there > - The durable ownership plan proves owners by identity through one shared prover, keyed by PID space > - This pull request records the PID space on the receipt and routes all three checks through one shared prover. It also judges the RC-50 copyback-owner hold on the shared path, and a proven-gone owner wakes its waiting recheck > - The benefit is that a takeover can never be authorized by the wrong host's PIDs, and a hold whose owner is provably gone is rechecked now rather than after its backoff ## Linked Issues or Issue Description Refs #19831 (plan, Phase 2 sections A and C), #19838 (inventory: "Copyback-owner hold (RC-50)", "Native copyback owner receipt", and the "One fact, many proofs" row `proveLocalWorkspaceOwnerStopped`). This follows #19997, which left `proveLocalWorkspaceOwnerStopped` alone. What happened? The receipt recorded `hostname` but no PID space, so every check compared hostnames and then probed the PID in this PID space. The registry could not judge the RC-50 hold (`processSpace: null`), so a hold whose owner was dead stayed on its backoff. Expected behavior Receipts carry `hostHash`. `proveCopybackOwnerGone` judges the owner in that space; another space is unprovable (`foreign_host` / board attestation). Receipts written before this change keep the previous same-hostname rule. The hold's owner is judged by the shared prover, and `gone` makes its waiting recheck due now. ## What Changed - `native-workspace-owner-stop-proof.ts`: new `copybackOwnerStamp`, `proveCopybackOwnerGone` and `copybackOwnerOnAnotherHost`. `proveLocalWorkspaceOwnerStopped` uses the shared prover. Its process-group orphan check is kept unchanged, because copyback children are a separate proof. - `native-workspace-finalization-ownership.ts`: the receipt records `hostHash`, and the takeover uses `proveCopybackOwnerGone` in place of the hostname compare plus `evaluateNativeControllerTakeover`. - `native-workspace-owner-recovery.ts`: the board confirmation refuses a live or locally unverifiable owner. An owner in another PID space stays the board's to attest, and `hostHash` must match the recorded receipt. - `workspace-recovery.ts`: new `wakeCopybackOwnerRecheck`. It makes a waiting `copyback_stop_unconfirmed` recheck due now (respecting provider cooldown and the deadline), grants no attempt, and only applies while the hold still names the proven owner. - `recovery/ownership-registry.ts`: `copyback_owner_hold` uses the shared stamp and registers a settle that wakes the recheck. It never resolves the hold itself. - The settling-kinds list in `ownership-reconciliation-report.test.ts` is updated. ## Verification - `native-workspace-owner-restart-recovery.test.ts`: 18 pass. Three new tests fail on `master` and pass here: - the receipt records its PID space; - a same-hostname receipt from another PID space is never judged by local PIDs: the takeover and the stop proof both refuse, and the board can still attest; - the RC-50 hold is judged `pid_dead` and its recheck is woken with no attempt spent, while a receipt with no PID space stays unprovable. The guard "a stamped owner in this PID space proven gone resumes copyback" passes on both. - `native-workspace-finalizer.test.ts` (65), `workspace-recovery-preparation-gates.test.ts` (47), `native-workspace-finalization-work.test.ts` (25) and `ownership-reconciliation-report.test.ts` (25) pass. - The server package typecheck passes. ## Risks - Receipts written before this change behave as before: the same-hostname rule, judged in this PID space. - The board confirmation path is unchanged for owners on another host. For an owner on the same hostname but in another PID space, it now asks the board instead of trusting local PIDs. - The hold settle only reschedules an existing recheck. The worker still proves the copyback process group before taking ownership. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-08 05:32:49 -06:00
DottaandClaude Opus 5.5	b6f58dc6e2	feat(recovery): settle runtime-control operations of proven-gone owners on the shared path (#20016 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A managed runtime control (start, stop, restart, repair, run) on an execution workspace writes a `running` operation row that blocks every other control on that workspace > - If the server that ran it crashes, the row stays `running` until the next control request sweeps it. The sweep had its own `kill(pid, 0)` check, with no process start time, so a reused PID looked alive until 60 s of heartbeat silence passed > - The durable ownership plan proves owners by identity through one shared prover and settles them from one reconciler > - This pull request stamps the owner's boot id and start time, judges it with `proveOwnerGone`, and registers a settle for the kind > - The benefit is that a workspace stranded by a crashed server frees itself in the next reconciler pass, and a reused PID no longer holds it ## Linked Issues or Issue Description Refs #19831 (plan, Phase 2 sections A and C), #19838 (inventory: "Managed runtime-control operation", and the "One fact, many proofs" table row `ownerProcessIsAlive` workspace-operations.ts). What happened? `workspace_runtime_control_operation` was a report-only kind with a bespoke liveness check: `ownerProcessProvenGone` probed the PID with signal 0 and had no start time. A server restart that reused the PID kept the operation "alive" until its heartbeat aged out, and nothing settled the row until another control request arrived. Expected behavior The owner is proven by the shared prover: a dead or reused PID in the stamp's PID space is gone. Another host, a stamp without a PID space, or a live PID without a recorded start time stays unprovable and keeps the heartbeat lease as its fenced takeover at the next control. ## What Changed - `workspace-operations.ts`: - The owner stamp adds the server's `bootId` and `processStartedAt` (`currentRuntimeControlOwner`, no I/O). - The new `runtimeControlOperationOwnerStamp` is shared with the registry. - The sweep proves the owner with `proveOwnerGone`. `ownerProcessProvenGone` is deleted. - The terminal CAS and log append are factored out (`claimRuntimeControlTermination`, `appendRuntimeControlReconciliationLog`). - New `settleRuntimeControlOperation` for the reconciler: it skips operations this process is driving, requires the same owner as the proof, CAS-fails the row and writes a `workspace_operation.reconciled` activity entry. - `recovery/ownership-registry.ts`: the kind uses the shared stamp and registers the settle. - The settling-kinds list in `ownership-reconciliation-report.test.ts` is updated. ## Verification - `workspace-runtime-control-recovery.test.ts`: 21 pass. The 6 new tests are in "shared ownership path". 5 fail on `master` and pass here: - a reused PID is recovered at once; - the reconciler settles a dead owner with activity and frees control; - an unprovable owner is never settled, and a read-only pass changes nothing; - a re-stamped owner is kept; - new operations carry the boot id and start time. The live-owner guard passes on both. - `ownership-reconciliation-report.test.ts` (25), `workspace-operations-reconciliation.test.ts` (9) and `execution-workspace-runtime-control-conflict.test.ts` (24) pass. - The server package typecheck passes. ## Risks - Rows written before this change have no start time. A live PID on such a row stays unprovable, and the 60 s heartbeat lease still applies at the next control. A dead PID is still proof. - The reconciler now terminalizes these rows. It does so only on `gone`, and only with the same owner and an `updated_at` CAS. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-08 05:07:58 -06:00
DottaandClaude Opus 5.5	2aac7d29ff	feat(recovery): settle workspace runtime leases of proven-gone owners on the shared path (#20015 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A task that starts, stops or repairs a workspace's runtime services takes a runtime lease, so two tasks never drive the same workspace at once > - A lease whose owner task or run has ended must be released. Until now that happened only when the next claimant arrived and checked, or when 30 minutes of inactivity passed > - The durable ownership plan releases a record when its owner is proven gone, by identity and never by age, through the shared reconciler > - This pull request registers the runtime lease with the reconciler. Proof releases a lease; the 30-minute idle rule stays only as a fairness policy at claim time > - The benefit is that a workspace locked by a finished task frees itself at once, and a live task's lease is never taken by age ## Linked Issues or Issue Description Refs #19831 (plan, Phase 2 sections A and C), #19838 (inventory: "Execution workspace runtime lease", and the "Released only by TTL/age" list). What happened? `execution_workspace_runtime_lease` was a report-only kind. A lease whose issue or run had ended stayed held until another claim came, or until its 30-minute TTL expired. The claim path also mixed the TTL with the ownership check. Expected behavior An issue- or run-owned lease is released when its owner is proven gone: the issue is missing, hidden or terminal, or the run is missing or terminal. Age is not proof. An agent-scoped owner has no identity to prove, so the lease's expiry plus the next claim's locked takeover is the fenced handoff. ## What Changed - New `workspace-runtime-lease-settlement.ts`: `settleExecutionWorkspaceRuntimeLease` locks the row, requires the same owner key, re-proves that owner gone, and only then deletes the lease (compare-and-swap). - `workspace-runtime-leases.ts`: the new `proveWorkspaceRuntimeLeaseOwnerGone` holds the identity proof. The claim path checks it first, then applies the 30-minute idle reclaim as a separate fairness policy. - `recovery/ownership-registry.ts`: the kind registers the settle. Owners are now `issue`, `run`, or a fence on `expires_at` for agent-scoped keys. `RecordOwner` gains an `issue` kind. - `recovery/ownership-invariant-report.ts`: issue owners are classified (missing, hidden or terminal means gone). Salvaged from the stopped Phase 2 workspace agent's uncommitted work. I resolved it against current `master` and verified it. ## Verification - New `workspace-runtime-lease-ownership.test.ts`: 8 pass. Six fail on `master` and pass here: - a crash after the claim; - hidden or missing issues, and terminal or missing runs; - an agent-scoped lease never settles; - a read-only pass changes nothing; - a lease taken over after the proof is kept; - the release reason names the ownership proof, not the idle policy. Two guards pass on both: a live owner is never released by age, and the idle reclaim still works at claim time. - `ownership-reconciliation-report.test.ts` (25), `workspace-runtime-leases.test.ts` (10) and `execution-workspace-runtime-lease-route.test.ts` (4) pass. - The server package typecheck passes. ## Risks - The reconciler now deletes leases whose owner is proven gone. It always re-proves under the row lock against the exact owner key. - The idle reclaim behaves as before at claim time. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-08 04:45:48 -06:00
DottaandClaude Opus 5.5	ada44a669a	feat(recovery): settle stale issue execution and checkout locks on the shared ownership path (#20006 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A task is locked to the run that executes it, and the lock must be released when that run ends > - A restart between the run's terminal write and the lock release left the task locked to a dead run until a periodic sweep noticed > - The durable ownership plan moves each in-progress record onto the shared reconciler, which settles a record in its first pass after its owner is proven gone > - This pull request registers the task execution and checkout locks with the reconciler, and makes the lock release promote any messages saved behind it > - The benefit is that a task stuck behind a dead run is freed in the first reconciler pass, and its waiting messages go out at once ## Linked Issues or Issue Description Refs #19831 (plan, Phase 2 sections A and C), #19838 (inventory: "Issue execution lock" and "Issue checkout lock"), #19956 (work-intent outbox), #20004 (lost-process reapers). What happened? The task locks were report-only kinds in the ownership registry. A lock whose run had already ended was released only by `sweepStaleIssueLocks` on its own schedule. Messages saved behind the lock then still waited out their 30-second refusal backoff. Expected behavior A lock whose owner run is proven ended (a terminal or missing run row) is settled by the reconciler on that proof, never by age. A lock that a same-run resume, an explicit-retry claim, a manual admission hold or a newly live run still needs is kept. ## What Changed - New `server/src/services/recovery/issue-lock-settlement.ts`: `settleIssueLock` runs the stale-lock sweep for exactly that task, so its existing compare-and-swap clear and all of its guards decide. It loads the sweep lazily. - `recovery/ownership-registry.ts`: `issue_execution_lock` and `issue_checkout_lock` register `settleIssueLock`. - `recovery/service.ts`: clearing a stale lock now calls `promoteDeferredWorkIntents(trigger: "lock_released")` before its write, in the same transaction. - The periodic sweep stays, because its pre-pass also hands lost-process runs to the single reaper (#20004). The reconciler is now the faster backstop for the lock itself. ## Verification Each focused file ran alone, in one process, on embedded Postgres: - `ownership-reconciliation-report.test.ts` (25 pass). A new test covers dead, live and resuming locks: a read-only pass never settles; the dead run's lock is cleared with an `issue.stale_lock_cleared` row; the live and same-run-resume locks are kept. It fails against `master`'s registry and passes here. The test that lists settling kinds now includes both lock kinds. - `recovery-stale-issue-lock-sweep.test.ts` (49 pass). A new test checks that a message saved behind a stale lock is made due when the lock is cleared. It fails with `master`'s `service.ts` and passes here. - The server package typecheck passes. ## Risks - The reconciler can now clear a lock that only the periodic sweep cleared before. It goes through the same compare-and-swap and guards, so the outcome is the same, only sooner. - More passes call the sweep for a single task. Each call is scoped to one issue. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-08 03:51:18 -06:00
DottaandClaude Opus 5.5	7c508d3e42	fix(recovery): collapse the two lost-process reapers onto one stamp and the shared owner prover (#20004 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When the control plane restarts, it must decide whether each running agent's process is gone before it releases the run > - Two reapers made that decision for legacy runs, with different rules and different outcomes. One also waited 5 minutes even when it could already prove the process was dead (RC-27) > - The durable ownership plan moves every such decision onto one prover, `proveOwnerGone`, and never releases work by age alone > - This pull request collapses the two reapers onto one owner stamp and the shared prover, removes the age wait for a proven-dead process, and never releases a process it cannot prove gone > - The benefit is that a lost process ends one way, at once, with its normal retry, and a process on another host is never released by mistake ## Linked Issues or Issue Description Refs #19831 (plan, Phase 2 sections A and C), #19838 (inventory: "One fact, many proofs" and the age-only list), #19871, #19883, #19953, #19997. What happened? - `recoverActiveExecutions` (heartbeat) and `terminalizeOrphanedRunningRun` (recovery service) both judged a lost legacy child. They used different liveness rules (EPERM counted as alive in one, dead in the other) and gave different outcomes: `failed/process_lost` with a retry in one, `interrupted/orphaned_running_run` with no retry in the other. - The active-execution reaper kept a 5-minute age guard even when the child was provably dead. - Hot-restart adoption judged the child with raw PID checks. Expected behavior One prover decides. A child that is proven gone is settled at once, through one path with one outcome. An unprovable child (another PID space, a sandbox, or a pre-#19871 row on an ephemeral server) is never released, and Stop is its operator action. ## What Changed - New `server/src/services/run-process-owner.ts`: `lostProcessOwnerStamp` and `proveRunProcessGone`. They wrap the shared `runProcessOwnerStamp` (#19997) with the one-release legacy fallback (#19883) and the group-leader case. - `heartbeat/recovery.ts`: - The active-execution reaper judges a legacy child with the shared prover. A child proven gone skips the age guard; master's exemption for a proven-gone controller (#19953) is kept. - An unprovable child gets the detached marker, whose message says to stop the run. - The legacy host fallback is recorded as `LEGACY_HOST_FALLBACK_EVENT`. - `heartbeat/recovery.ts` (hot-restart adoption): a run is marked lost only on a `gone` proof. An unprovable child stays adopted and held. - `recovery/service.ts`: the stale-lock sweep no longer judges a legacy child. It hands the run to `settleLostRunProcess`, which is the active-execution reaper wired in `heartbeat.ts`, and then reads the outcome back. Native runs keep the sweep's own backstop, now judged with the shared prover. Salvaged from the stopped Phase 2 recovery agent's uncommitted work, which had no tests. I completed and corrected it: the salvaged adoption marked unprovable children lost, which risked duplicate execution. ## Verification - `recovery-stale-issue-lock-sweep.test.ts` (51 pass). Three new tests fail on `master` and pass here: - "hands a legacy run whose child is proven gone to the one lost-process reaper"; - "settles a legacy run whose child is proven gone without waiting out the age guard"; - "never releases a legacy run whose child is unprovable, and names Stop as the way out". Updated tests: - The PID-space test now expects `failed` for the one legacy outcome. - The explicit-retry race now injects `settleLostRunProcess`. - The two tests of the sweep's own terminal write use native runs, which keep that path. - `heartbeat-process-recovery.test.ts`: 418 of 419 pass. The one failure, "redacts opaque environment-bound credentials from Sentry diagnostics", also fails on unchanged `master`. - `heartbeat/recovery.test.ts` (97), `ownership-reconciliation-report.test.ts` (24), `native-remote-retry-admission.test.ts` (92) and `heartbeat-provider-limit-wait.test.ts` (4) pass. - The server package typecheck passes. ## Risks - Behavior change: a legacy run whose child died, when the orphan sweep finds it, now ends `failed/process_lost` with the reaper's retry policy, instead of `interrupted/orphaned_running_run`. - A pre-#19871 row on an ephemeral server, or a child in another PID space, now stays `running` with a detached marker until Stop. It is no longer released by age. - Not changed: a dispatched legacy run that recorded no PID and has no controller stamp keeps its lease-expiry plus age fence. Rows written since #19953 are judged by the controller proof. Older rows, and remote handoff, keep the fence. A different host is never proof, so removing the fence would strand runs after an ephemeral replacement. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-08 03:50:34 -06:00
DottaandClaude Opus 5.5	a4dbde6181	refactor(recovery): judge native owner processes with the shared prover (#19997 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs record the server and adapter processes that own them, and recovery must prove an owner gone before it takes over or releases its work > - Phase 2 of the durable ownership plan gave the repository one prover, `proveOwnerGone`. The native runtime still had four private proofs of the same fact > - The private proofs disagreed with the shared rules in edge cases: some judged a PID from another PID space, and one accepted an inconsistent start time as proof of reuse > - This pull request moves those four proofs onto `proveOwnerGone` and deletes the private liveness code > - The benefit is one rule for "the owner is gone" across native recovery, with the stricter shared behaviour where they differed ## Linked Issues or Issue Description Refs #19933, #19950, #19953 (the shared ownership path and earlier Phase 2 kinds). What is the problem? Native recovery proved dead owners with four private checks instead of the shared prover. The checks used different rules for PID spaces, reused PIDs and unreadable start times. What should happen? Every native owner proof uses `proveOwnerGone`. Only a `gone` verdict releases or takes over. The remote fence and the one-release legacy fallback stay as they are. Why does it matter? Recovery decisions about the same process must agree. A dead PID in another PID space must never be used as proof. ## What Changed - `native-restart-recovery.ts` `evaluateNativeControllerTakeover`: the local-process branch now calls `proveOwnerGone`. The remote fenced branch (#19830), the `operator_attested` sentinel and the null-host fallback (`processSpaceUnrecorded` with `legacyHostFallback`, recorded by callers as `legacy_host_unknown_dead_pid_proof`) are kept. Reasons are unchanged, plus a new `controller_pid_start_inconsistent`. Owners built from receipts with no PID space are judged in this PID space: their callers check the recorded host first. - `native-failed-run-owner-release.ts`: `recordedProcessMayBeLive` is deleted. `runProcessMayBeLive` and `controllerMayBeLive` release only on a `gone` verdict. Each use of the legacy fallback is recorded as `legacy_host_unknown_dead_pid_proof` when the release commits. - `native-session-executor.ts`: `verifiedRecoveryProcessIsAlive` is replaced by `recoveryProcessMayBeAlive`, which is false only when the prover proves the runner gone. Signalling a recovered runner requires a proven same process. - `native-local-process-stop.ts` `recordedLocalExecutionStopped`: the bound identity (launch event, host, PID, group, start time) is judged by the prover. It uses `process_host_hash` when the row has one. - New `native-runtime/native-owner-stamps.ts`: controller and adapter-process stamp builders, shared with the ownership registry. Not changed: `proveLocalWorkspaceOwnerStopped` (workspaces), `evaluateNativeProviderProcesses`, the stop observation in `recordNativeLocalProcessStop`, and `runProcessMayBeLive` in `native-workspace-sync-retention.ts`. ## Verification - `cd server && npx vitest run src/services/native-runtime/native-restart-recovery.test.ts src/services/controller-process-identity.test.ts`: 51 passed. - `cd server && npx vitest run src/services/native-runtime/native-failed-run-owner-release.test.ts`: 14 passed. - `cd server && npx vitest run src/services/native-runtime/native-controller-host-recovery.test.ts`: 2 passed. - `cd server && npx vitest run src/services/quarantined-task-retry.test.ts -t "keeps quarantine"`: 18 passed. `explicit-native-continuation.test.ts`: 414 passed. `ownership-reconciliation-report.test.ts`: 23 passed. - Red with the master versions of the two modules: 3 new tests fail ("accepts a reused PID only when the live process started definitely later", "never releases on a dead PID recorded in another PID space", "keeps the one-release fallback for a run process saved before its PID space, and records it"). The tests were written after the code. - Two fixtures changed to match the shared rules: - `controller-process-identity.test.ts`: the "recycled" case recorded a start after the live process, which is an inconsistent stamp. It now records an earlier start (a real reuse), and the inconsistent case is asserted as alive. - `quarantined-task-retry.test.ts` `worker_alive`: the live worker now records this process's real start time. The old fixture was a reused PID, which the prover correctly proves gone. - `cd server && npx vitest run src/services/native-local-process-stop.test.ts`: 5 passed (review round 1). With the master module, 2 fail: the other-PID-space case and the reused-PID case. - Not run locally: typecheck (CI runs it) and the wide suites. ## Risks Behaviour changes, all from the shared rules: - A live PID observed to start before the recorded controller is no longer accepted as recycled (`controller_pid_start_inconsistent`; the run waits for evidence). - A failed-run owner whose recorded adapter process or controller is in another PID space, or is a remote sandbox PID, is no longer released by a local probe. It keeps the lease path. - An unreadable start time is no longer exit evidence for a recovered local runner. - A local execution whose PID was reused by a later process now counts as stopped. Before, any live PID blocked the proof. - A local execution whose row records another PID space is no longer counted as stopped by a dead local PID. - A PID space that cannot be read here makes a hostless receipt owner unprovable. No persisted state changes. Rows saved by the previous release keep the one-release null-host fallback, as before. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, local test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-08 02:35:15 -06:00
DottaandClaude Opus 5.5	8475826d2d	fix(wake-queue): deliver user comments whose wake a restart lost or a stale continuation dropped (#19992 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A user's follow-up comment on a task must reach the agent exactly once, through restarts (invariant 4) > - The final unmasked local chaos storm (96 tasks, 31 restarts) passed 94 tasks. The 2 failures were both lost user comments > - In t69, a restart killed the server between saving the comment and writing its wake. In t72, a comment's own run was cancelled as stale, because the earlier run completed the task first without reading the comment > - This pull request adds a reconciler rule that gives a lost comment its wake, and saves the comment before a stale cancellation > - The benefit is that a user's message is never silently dropped when a restart or a completion race hits at the wrong moment ## Linked Issues or Issue Description Refs #19956 (work-intent outbox), #19944, #19831 (plan, invariant 4). What happened? In chaos wave `w-20261108T055444-s9401` (local, unmasked, codex and claude, 30-min K13 storm, master `46bf913fb9`), 94 of 96 tasks passed. Two failed: - t69, R1 `rested_not_done`. The user's comment was saved at 06:14:59.358 and reopened the task to `todo`. A K1 SIGKILL fired 28 ms later, before the wake was written. No wake row exists. Recovery skips a `todo` task whose last run succeeded, so the task sat with no run until the wave ended. - t72, R4 `concurrent_runs`. A comment posted during a run queued its own run. That run was claimed at 06:16:04.874. The first run's completion marked the task `done` 2 ms later. The comment's run was then cancelled with `continuation_task_ownership_changed` before dispatch, and the comment was lost. Expected behavior Each accepted user comment is delivered once, unless the task is cancelled or reassigned, or a person closes it. Steps to reproduce `node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --replay w-20261108T055444-s9401/t69` (or `.../t72`). ## What Changed - `recovery/ownership-invariant-report.ts`: a new reconciler-owned I4 rule, `user_comment_without_wake`. It finds a user comment that meets all of these conditions: - it is newer than every run of the task; - the task is assigned, is not a conversation, has no live run or lock, and has at least one earlier run; - no wake request carries the comment. A comment younger than 30 s is not reported yet, because it may still get its normal wake. Its end-of-grace time is returned as `nextDueAt`. - `native-runtime/native-finalization-reconciler.ts`: the backstop step is now exported as `applyOwnershipReportBackstops`. For each such comment it saves the missing outbox receipt (`executionWait: wake_lost`) and promotes it. - `work-intent-outbox.ts`: new `saveUndeliveredUserComments`, which the reconciler rule and the stale-run path both use. It saves only live user comments that no completed run received. It writes nothing for a task that is reassigned, is a conversation, or has a status the caller does not allow. - `heartbeat/queue.ts`: the saved-receipt pass that already sends pause-saved messages also sends `wake_lost` receipts. It uses the same per-comment consumption check and the same chat-input exclusion. - `heartbeat/stale-comment-continuation.ts` and `heartbeat.ts`: before a stale continuation is cancelled, its undelivered user comments are saved as a deferred receipt. The release drain then reopens the `done` task and delivers them once. This applies only when an agent run that started before the comment completed the task after it. A task that a person marks done is left closed. - `tests/restart-chaos/grader.ts`: R4 ignores runs cancelled before dispatch (`continuation_task_ownership_changed`) when it looks for overlapping execution. Those runs never executed. Comment consumption still counts them. ## Verification Each focused file ran alone in one process (`--maxWorkers=1`): - `heartbeat-held-comment-wake.test.ts` (22 pass). This includes two new cases: - t69 shape: the reconciler rule fires, the backstop delivers the comment once, and a second pass does nothing; - t72 shape: the stale run is cancelled, the task reopens, and the comment is delivered once. Both fail with the rule or the save disabled, and pass with them. - `ownership-reconciliation-report.test.ts` (23 pass). It includes the rule's boundaries: a carried comment, a too-young comment (deadline only) and an already-read comment. - `tests/restart-chaos/grader.test.ts` (17 pass). The new R4 case fails with the old grader. A cancelled run that is not a pre-dispatch cancellation still fails R4. - Unchanged and passing: comment wake batching (27), refused assignment wake (3), wake-queue postgres adapter (103), stale queue invalidation (68), deferred receipt work (45). The first version also reopened a task that a person had marked done, which broke one deferred-receipt test. That case is now excluded. - The server package typecheck and the restart-chaos harness typecheck pass. - Same-session chaos context on master: K6 crash loop `w-20261108T063326-s9402`: 6 of 6 pass. The K4 long-hold wave is still running. After merge, I'll replay only the targeted fault shapes (t69 and t72). ## Risks - The reconciler rule runs on each report pass. It is bounded by the 1-hour lookback and the per-kind row cap, and it excludes conversation tasks, so chat input keeps its own authorization. - A comment younger than 30 s waits up to the end of its grace period before the backstop acts. - The stale-run save reopens a `done` task only for a comment that the completing run could not have read. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-08 02:14:45 -06:00
DottaandClaude Opus 5.5	dc0534d1b5	feat(recovery): destroy a retained orphan sandbox once nothing references it (#19976 ) <!-- Write all pull request text in Simplified Technical English (ASD-STE100): short sentences, one instruction per sentence, simple approved vocabulary, and the active voice. --> ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agent runs use remote sandboxes held by environment leases > - #19964 settles an orphaned active lease when its holder is proven gone. It stops and keeps the sandbox, because recovery may still need its files > - Nothing destroyed that retained orphan later, so a stopped sandbox could stay at the provider indefinitely > - This pull request destroys a retained orphan once no recovery, preservation or retention obligation references it, from the reconciler pass only > - The benefit is that orphaned sandboxes do not leak, and a sandbox that recovery still wants is never destroyed ## Linked Issues or Issue Description Refs #19964 Refs #19933 What happened? After #19964, an orphaned sandbox is stopped and kept (`released` with a stop-and-retain receipt). No path destroyed it when recovery no longer needed it. Expected behavior A retained orphan is destroyed once database facts show that no obligation references it. It is never kept past the environment's retention window (the provider-attested lease `expiresAt`) only because of a soft reference. A sandbox that an active operation or a live recovery path still needs is never destroyed. Steps to reproduce 1. Let a run's server die after the run is terminal and before its sandbox lease is released. 2. Let the ownership settle stop and keep the sandbox. 3. Resolve the run's recovery. The stopped sandbox stays at the provider. ## What Changed - `sandbox-stop-and-retain.ts`: an ownership settle's stop intent also writes a top-level `orphanedRunLeaseRecovery` marker. The marker stays after the stop intent is removed. - `environment-retained-orphan-settlement.ts` (new): the settle for `environment_retained_orphan_sandbox`. It locks the lease row and checks database facts: - the run is terminal; - always held: native export resume, stop-and-retain intent, in-flight cleanup claim, workspace restore recovery, workspace recovery cleanup, bound source-recovery consumer, pending legacy shutdown fence, or another live lease on the same provider resource; - held while the stop receipt is a usable source (`expiresAt` not passed): pending native export or preservation fence, open recovery action on the issue, or a queued, scheduled or running run on the issue. - Then it moves the lease to `pending_cleanup` (ephemeral) with a compare-and-swap. The pending-cleanup sweep destroys it from the recorded lease. The settle never calls a provider. - `recovery/ownership-registry.ts`: the kind lists only releasable retained orphans (an SQL mirror of the checks), so a held orphan is never polled. It reports `nextDueAt` = the earliest retention-window end of a softly held orphan. - `recovery/ownership-invariant-report.ts`: an `OwnableKind` may report `nextDueAt`; the pass folds it into the report's next due time, so the reconciler wakes when the window ends. - `environment-cleanup-state.ts`: the first destroy attempt of a released orphan is due at once. ## Verification - `cd server && npx vitest run src/__tests__/environment-retained-orphan-sandbox.test.ts src/__tests__/ownership-reconciliation-report.test.ts src/__tests__/environment-run-lease-ownership.test.ts src/__tests__/heartbeat-orphaned-active-lease-sweep.test.ts src/__tests__/heartbeat-pending-cleanup-sweep.test.ts` (embedded Postgres). - Red on master, green here (tests written after the code): the settle marks the lease; an unreferenced retained orphan is destroyed through the cleanup sweep; a softly held orphan waits until its window ends and the pass reports that due time. - Counter-cases (green): never destroyed with an open recovery action, a queued run, a scheduled retry, a stop-and-retain intent, a native export resume, a workspace restore recovery, an in-flight claim, a pending legacy shutdown fence, another live lease on the resource, or an active run; a read-only pass changes nothing; a retained sandbox without the orphan marker is left alone. - Full typecheck is left to CI (local memory limit). ## Risks - A retained orphan saved by #19964 before this change has no marker, so it is not destroyed by this path. The window was short (since #19964 merged). - A held orphan with no `expiresAt` is released on the first reconciler pass after its last reference clears. If that clearing writes no signal the reconciler wakes on, the release waits for the next pass. - The SQL listing mirrors the settle's checks except the pending native export, which the settle checks in code. Such an orphan stays on the settle watch until that export settles. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edit, embedded Postgres tests). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-08 01:38:22 -06:00
DottaandClaude Opus 5.5	838e9bf3d2	feat(recovery): settle sandbox creation intents of dead creators on the shared ownership path (#19978 ) <!-- Write all pull request text in Simplified Technical English (ASD-STE100): short sentences, one instruction per sentence, simple approved vocabulary, and the active voice. --> ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Before it asks a provider to create a sandbox, the server writes a creation intent that names its creator process > - If the creator dies during the provider call, the intent waits for a 30-minute settle deadline before recovery reconciles it with the provider > - After a restart, creation recovery made such intents due early with its own process check (`promoteAbandonedSandboxCreations`) > - Phase 2 of the durable ownership plan moves each in-progress record kind onto one shared owner proof and deletes its bespoke check > - This pull request registers a settle for `sandbox_creation_intent` and calls the shared path where creation recovery called the bespoke promotion > - The benefit is one proof for every kind, while the provider absence proof and the cross-host handoff stay unchanged ## Linked Issues or Issue Description Refs #19964 Refs #19933 What happened? Sandbox creation recovery had its own liveness check (`controllerProcessExited`) for the creator of a pending intent. Each kind with its own check can drift from the shared rules (PID space, PID reuse, current boot, deployment replacement). Expected behavior The shared prover judges the creator. On gone, the intent becomes due now and its creator is marked lost. Provider absence stays inconclusive until the settle deadline. A creator on another host is never proven gone; its intent keeps the deadline. Steps to reproduce 1. Start a sandbox acquire so a creation intent is written. 2. Kill the server before the provider reply is committed. 3. Restart. The intent must become due through the shared ownership path, not a separate check. ## What Changed - `sandbox-creation-intent-settlement.ts` (new): compare-and-swap settle on the exact creator and a deadline that is still in the future. It sets `nextAttemptAt` to now, marks `creationControllerLost` with the proof reason, and wakes creation recovery in the same transaction. - `recovery/ownership-registry.ts`: `sandbox_creation_intent` registers the settle. An intent whose creator is already lost is no longer listed; it belongs to the creation recovery deadline. - `sandbox-creation-operations.ts`: creation recovery calls `settleOwnableKind(db, "sandbox_creation_intent")`. Deleted `promoteAbandonedSandboxCreations` and its use of `controllerProcessExited`. ## Verification - `cd server && npx vitest run src/__tests__/sandbox-creation-intent-ownership.test.ts src/__tests__/sandbox-creation-operations.test.ts src/__tests__/ownership-reconciliation-report.test.ts` (embedded Postgres). - The existing lost-creator matrix stays green on the shared path: destroy on the next sweep; page past unprovable creators; issue-bound creation after a later run; reused PID; live creator here; live creator in another process; creator on another host; early lookup finds nothing yet (absence inconclusive); committed lease untouched. - New, red on master and green here (written after the code): the reconciler pass makes a dead creator's intent due now and keeps the absence deadline. - New counter-cases (green): live creator, another host, and an intent without a creator keep their deadline; a read-only pass changes nothing. - Full typecheck is left to CI (local memory limit). ## Risks - Low risk. The settle writes the same state the deleted promotion wrote, plus the proof reason. - Previous-release rows: every creation intent already stored its creator with a PID space, so no fallback is needed. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edit, embedded Postgres tests). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-08 01:07:58 -06:00
DottaandClaude Opus 5.5	3ff62fc6db	feat(recovery): model an unclaimed pending-cleanup item as owned by the cleanup queue (#19980 ) <!-- Write all pull request text in Simplified Technical English (ASD-STE100): short sentences, one instruction per sentence, simple approved vocabulary, and the active voice. --> ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A sandbox that still needs teardown waits as a `pending_cleanup` environment lease until the cleanup queue retries it > - The ownership reconciler checks invariant I1: every in-progress record names an owner > - An unclaimed pending-cleanup item names no process by design, so I1 reported every one (61 in the Phase 2 data) > - That noise hides real ownership gaps > - This pull request models the item honestly: the cleanup queue owns it, with a visible next action at its saved retry deadline > - The benefit is that I1 measures real gaps, and a queue that stops acting shows up as an I7 gap instead of silence ## Linked Issues or Issue Description Refs #19933 What happened? `environment_pending_cleanup` listed every unclaimed pending-cleanup item with owner `none`, so I1 reported each as `no_owner_stamp`. Expected behavior The item names the queue that acts on it and its next due time. It is reported only when the queue is not acting on it (long past the deadline). No owner is invented, and nothing is released. Steps to reproduce 1. Let a sandbox teardown fail so its lease waits in `pending_cleanup`. 2. Run the ownership reconciliation pass. 3. See an I1 `no_owner_stamp` violation for a healthy queue item. ## What Changed - `recovery/ownership-registry.ts`: new owner kind `queue` (`queue`, `dueAt`, `graceMs`). `environment_pending_cleanup` names the environment cleanup queue. Its due time is the same retry deadline the cleanup worker sleeps until, including the at-once first attempt of a settled orphan. - `recovery/ownership-invariant-report.ts`: a queue owner is alive (`queue_scheduled`) while its deadline is current. When it is more than two retry intervals past, it is unprovable (`queue_overdue`) and reported as `I7_visible_next_action`. A queue is never proven gone and nothing is settled. - The listing reads the in-flight flag as text, so a malformed value cannot break it. ## Verification - `cd server && npx vitest run src/__tests__/environment-pending-cleanup-queue-owner.test.ts src/__tests__/ownership-reconciliation-report.test.ts` (embedded Postgres). - Red on master, green here (written after the code): a current item names the queue (no I1); a long-overdue item is an I7 `queue_overdue` gap. - Green on both: an in-flight item is not listed here (it is a teardown claim). - Full typecheck is left to CI (local memory limit). ## Risks - Low risk. Report only: no record is changed. - The grace (two retry intervals) is a reporting threshold, not a release rule. A worker that sleeps longer than that for a legitimate reason would show as an I7 gap. - `#19966` changes the same listing predicate to the same text comparison; a rebase of either resolves trivially. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edit, embedded Postgres tests). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-08 00:39:11 -06:00
DottaandClaude Opus 5.5	95f3cab0cd	fix(recovery): settle a workspace finalize attempt whose process died mid-copyback (#19985 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The durable-ownership plan settles in-progress records whose owner is proven gone, on one shared path. > - The crash-point suite (#19946) found two windows where a workspace finalize attempt stays `running` forever after a restart, even though its run succeeded through a later attempt. > - #19807's supersession covers only native-finalizer attempts. The local live-finalize path writes rows without that marker. > - This pull request stamps each finalize attempt's owner and settles an abandoned attempt on the shared path. > - The benefit is that no finalize row is left `running` with a dead owner, and the run's outcome never changes. ## Linked Issues or Issue Description Refs #19946 (crash-point suite), #19807 (RC-64 supersession), #19933 (shared settle path). What happened? At crash points `workspace.finalize.after_start_before_copyback` and `workspace.finalize.after_copyback_before_done` (run `cp-20261108T040253`, tasks t07 and t08), the run succeeded after the restart. The original `workspace_finalize` operation stayed `running`, and the ownership pass reported I2 (`owner_run_terminal`). What did you expect? The interrupted attempt is settled once its owner is proven gone, and the run outcome is left alone. Steps to reproduce 1. `node scripts/crash-points.mjs --point workspace.finalize.after_start_before_copyback`. Or run the new report test case below. ## What Changed - `workspace-operations.ts`: every `workspace_finalize` attempt records `finalizeOwner` (the process stamp, with no I/O) and is tracked in a process-local set while it runs (`isWorkspaceFinalizeOperationLive`). - `ownership-registry.ts`: `native_workspace_finalize_attempt` takes its owner from the stamp, or from the run for rows written before the stamp. It now registers a settle. - New `recovery/workspace-finalize-attempt-settlement.ts`: a compare-and-swap that marks the attempt `failed` with `abandonedFinalization` provenance and logs a `workspace_operation.reconciled` activity. It skips an attempt live in this process, an uncertain export and a save-work record. It does not change the run. ## Verification - `cd server && npx vitest run src/__tests__/ownership-reconciliation-report.test.ts`: 19 pass. The new case covers a dead stamped owner (settled), a current-process owner (kept), another PID space (kept), an uncertain export (kept), a pre-stamp row of a terminal run (settled) and of a running run (kept), a read-only pass (no change), and the run status unchanged. - The finalize-related suites pass (100 tests in 5 files): `heartbeat-workspace-finalize-branch`, `native-workspace-finalization-work`, `workspace-operations-reconciliation`, `offline-workspace-operation-reconciliation`, `finalized-workspace-recovery-migration`. - Red/green: the new case fails on master, where the attempt stays `running`. I wrote the test after the code. - Crash-point re-proof with `--point` for both windows: to be run once the harness slot is free, and reported on this PR. ## Risks - Fail-open risk: settle needs a `gone` proof from the shared prover, and another PID space is never proof. It re-checks the stamp in the transaction and skips attempts this process is running. - Previous-release story: rows without `finalizeOwner` are owned by their run, and they settle only when that run is terminal. - Nothing runs before `turn.start`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-08 00:17:02 -06:00
DottaandClaude Opus 5.5	bf0d45a4d2	feat(recovery): settle teardown claims of dead claimants on the shared ownership path (#19966 ) <!-- Write all pull request text in Simplified Technical English (ASD-STE100): short sentences, one instruction per sentence, simple approved vocabulary, and the active voice. --> ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A sandbox teardown writes a claim on its environment lease that fences other cleanup for 15 minutes > - After a restart, the cleanup sweep ended the fence of a claim whose claimant had exited, with its own process check (`releaseLostPendingCleanupClaims`) > - Phase 2 of the durable ownership plan moves each in-progress record kind onto one shared owner proof and deletes its bespoke check > - This pull request registers a settle for `environment_teardown_claim` and calls the shared path where the sweep called the bespoke release > - The benefit is one proof for every kind: a local claimant proves death by its stamp; another host stays a fence handoff ## Linked Issues or Issue Description Refs #19965 Refs #19964 Refs #19933 What happened? The teardown claim had its own liveness check (`controllerProcessExited`) in the pending-cleanup sweep. Each kind with its own check can drift from the shared rules (PID space, PID reuse, current boot). Expected behavior The shared prover judges the claimant stamped for this exact attempt. On gone, the fence ends now and a counted sweep attempt is refunded once. A claim held as a fence (another claimant or another host) waits for its deadline. Steps to reproduce 1. Start a sandbox teardown so the lease holds an in-flight claim. 2. Kill the server. 3. Restart. The claim must be released by the shared ownership path, not by a separate check. ## What Changed - `environment-teardown-claim-settlement.ts` (new): compare-and-swap settle on the exact attempt and claimant. It ends the fence, refunds a counted attempt, and is single-use. It never acts on a fence owner. - `recovery/ownership-registry.ts`: `environment_teardown_claim` registers the settle. - `heartbeat/recovery.ts`: the sweep calls `settleOwnableKind(db, "environment_teardown_claim")`. Deleted `releaseLostPendingCleanupClaims` and its use of `controllerProcessExited`. - `environment-cleanup-state.ts`: the attempt cap, attempt key and their SQL readers move here, so the settle and the sweep share one reader. - `heartbeat-pending-cleanup-sweep.test.ts`: the overlap barrier now holds the shared prover instead of the deleted check. ## Verification - `cd server && npx vitest run src/__tests__/heartbeat-pending-cleanup-sweep.test.ts src/__tests__/environment-lease-owner-stamps.test.ts src/__tests__/ownership-reconciliation-report.test.ts` (embedded Postgres). - The existing lost-claim matrix stays green on the shared path: retry now; no failed-attempt charge for a dead server; one refund when sweeps overlap; keep a live claim; keep a newer claim that kept an older record; keep another host's claim; keep a claim without identity; page past fenced claims. - New, red before and green here (written after the code): the reconciler pass ends a dead claimant's fence and refunds its counted attempt. - New counter-cases (green): a read-only pass never settles; a live claimant, another PID space, and an unstamped fence claim keep their fence. - Full typecheck is left to CI (local memory limit). ## Risks - Low risk. The settle keeps the previous release semantics (fence ends, lease stays `pending_cleanup` and in flight until the sweep claims it again). - The stop-only claim still does not renew its fence; a live stop that runs longer than 15 minutes can still be taken over. This change does not alter that. - Stacked on #19965 and #19964. Merge those first. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edit, embedded Postgres tests). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 23:48:43 -06:00
DottaandClaude Opus 5.5	d33b3ac263	fix(wake-queue): save temporarily refused work intents and promote them through one path (#19956 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Users send work to agents as assignments, comments and Retries. The wake queue delivers each one as a run > - When an admission check refused a wake for a temporary reason, the wake often ended as a terminal skip, so the work was lost > - Restart-chaos waves found dropped comments and Retries (RC-37, s7532) and double delivery (RC-53). Each fix covered one hold type > - The durable ownership plan (section E) asks for one outbox rule: a closed set of terminal outcomes, a saved deferral for every temporary refusal, and one promotion path > - This pull request adds that rule. It saves messages refused by a pause, promotes deferred intents from every hold-release event through one function, and adds a reconciler backstop > - The benefit is that an accepted message is delivered exactly once after a pause lifts, and the reconciler reports any intent that is stuck or dropped ## Linked Issues or Issue Description Refs #19831 (plan, section E), #19838 (inventory, wake-queue rows and section 1b), #19927 (Phase 2 data: the I4 false positive), #19702, #19741, #19829, #19826. Builds on #19944 (section D, merged). What happened? - A user commented while the agent was paused, or while the company was paused. Admission refused the wake. Nothing saved the message as pending, so when the pause lifted, nothing delivered it. - Hold-release events woke different subsets of workers. A saved intent then waited out a 30-second refusal backoff after its hold cleared. - The I4 reconciler check matched free text with a regular expression. Phase 2 found a false positive: a delivered retry whose error said "retrying once". Expected behavior An accepted work intent ends only as delivered, consumed by a later run, task cancelled, or hard stop. Every temporary refusal saves the intent as `deferred` with its reason and its release trigger. Every hold-release event promotes deferred intents through one function (invariant 4). Steps to reproduce Pause an agent. Post a user comment on its task. Resume the agent. On `master` the comment is never delivered as a wake. ## What Changed - New `server/src/services/work-intent-outbox.ts`: - `WORK_INTENT_TERMINAL_OUTCOMES`: the closed set of terminal outcomes. - `TEMPORARY_REFUSAL_TRIGGERS`: maps each temporary refusal code to its release trigger. - `promoteDeferredWorkIntents`: the one promotion function. It removes the 30-second refusal backoff from the matching deferred intents. It then wakes run dispatch, the saved-receipt worker and stranded-work recovery, and hints each saved intent of the issue. - `heartbeat/queue.ts`: the #19829 save-as-deferred block is now a shared helper. A user's message refused by a paused agent, a paused company, or a budget pause that is not a reached hard stop is saved with `executionWait: { reason, trigger, message }`. The caller still gets a 409. The message is saved only when the task's latest run is terminal, because the release drain promotes a saved message behind that run. A task that never ran keeps the message in the task, and stranded-work recovery starts the task once the pause lifts. A `pending_approval` agent and a task-tree hold are not changed here: the release drain still fails a promoted wake for a pending approval, and tree holds have their own resume path. - Hold-release events call `promoteDeferredWorkIntents`: - accounting settled (`run-cost-accounting.ts`); - environment lease released (`environments.ts`, two sites); - native run projected terminal (`native-run-projection.ts`); - budget pause lifted (`budgets.ts`); - company reactivated (`companies.ts`); - task-tree hold released (`issue-tree-control.ts`); - agent lifecycle ready (`agent-lifecycle-driver.ts`). - Reconciler (`recovery/ownership-registry.ts`, `ownership-invariant-report.ts`, `native-finalization-reconciler.ts`): - registers three new kinds: `wake_request_deferred`, `queued_run` and `scheduled_retry_run`; - I4 now reads the closed refusal code (the wake `reason`) of accepted intents, not free text; - a new I4 check reports a deferred pause refusal whose pause has lifted for more than 5 minutes. The worker then promotes it as a backstop; - a lifted pause inside its 5-minute grace window returns its deadline through `nextDueAt`, and the worker also wakes on agent-lifecycle signals (company and agent pause changes). So a missed release event is still checked. - `doc/SPEC-implementation.md`: adds the rule "Accepted work intents are delivered exactly once". ## Verification Each focused file ran alone in one vitest process (`--maxWorkers=1`) on embedded Postgres. Results at `2875c85fd2`: \| Question \| Loop \| Elapsed \| Result \| \|---\|---\|---\|---\| \| Is a comment refused by a pause saved, then delivered once after the pause lifts? \| `heartbeat-held-comment-wake.test.ts -t "pause lifts"` \| 8 s \| 2 pass. With `queue.ts` from `master`, both fail: no intent is saved \| \| Does I4 use the closed code, and does it catch a stuck deferral? \| `ownership-reconciliation-report.test.ts` \| 45 s \| 20 pass. This includes the Phase 2 "retrying once" false positive and the case "paused, then lifted" \| \| No regression on the paths that changed \| held-comment (15), archived-company guard (8), undispatched execution wait (32), deferred receipt work (45), recovery pause durability (16), native run projection work (55), tree-control routes and services (19, 1, 7) \| 8–334 s each \| All pass \| - Exactly-once: each pause test runs two promotions (a release event, then the reconciler backstop) and many dispatch passes. It asserts one delivery. The existing consumption tests (RC-53) still pass. - Server package typecheck passes. - Speed: nothing new runs before `turn.start`. The new admission work happens only on a refusal. Promotion runs in the release transaction and in the background reconciler. - Not run here: chaos waves. The coordinating session schedules them on the shared harness. ## Risks - `promoteDeferredWorkIntents` moves `updated_at` back for deferred intents that were refused in the last 30 seconds. A busy company with many release events can retry refused intents more often. Each retry re-runs normal admission, and the refusal resets the backoff. - More release sites now wake run dispatch and stranded-work recovery. These notifications are coalesced per company. - Lifting a pause is now the operator's explicit authorization to send what the pause refused, as Send is after a Stop. When the pause stopped the task's run, the lifted pause issues the task's next explicit wake, which adopts the saved message once. Other messages queued behind an acknowledged Stop keep the existing rule. If Stop recovery still holds the task, the message waits for that recovery. - An agent pause's stop sweep keeps the messages that the pause deferred (`preservePauseDeferred`). Terminate and cleanup stops still end them. - The backstop check reads only pause refusals (`company.inactive`, `agent.not_invokable`), because those holds can be checked exactly. Other deferred kinds keep their existing promotion paths. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 23:43:17 -06:00
DottaandClaude Opus 5.5	576ba29fff	feat(recovery): settle the legacy controller lease on the shared ownership path (#19953 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Legacy (non-native) runs hold a controller lease on `heartbeat_runs` while the server prepares and dispatches the adapter. The shared ownership reconciler now settles proven-gone owners (Phase 2 of the durable ownership plan) > - The lease stamped only a boot id. The reconciler could never prove its owner gone (`missing_identity` in the Phase 2 data), and recovery released it by age: lease expiry (60 s), then a 5-minute age guard on dispatched runs > - A hard-killed server's legacy run therefore stayed `running` for up to 6 minutes after the next boot > - This pull request records the controller process with the claim, registers a compare-and-swap settle, and makes live-run recovery use the shared prover instead of the age guard when the controller is proven gone > - The benefit is that a dead controller's run is settled at once, while a live, unprovable, or previous-release controller keeps today's behavior ## Linked Issues or Issue Description Refs #19933 (the settle hook, merged). Stacked on #19950 (migration 0446, registry) and targets its branch until it merges; then it retargets to master. What happened? After a hard kill, a dispatched legacy run stayed `running` until its controller lease expired and its last update was 5 minutes old. What did you expect? The next boot proves the old controller gone and fails the run as a lost process at once. Steps to reproduce 1. Start a legacy (non-native) run and let it dispatch the adapter. 2. Kill the server (SIGKILL). 3. Start the server. The run stays `running` for up to 6 minutes. ## What Changed - `heartbeat_runs.controller_process` (jsonb, migration 0447): `{bootId, pid, hostHash, processStartedAt}` of the claiming server process. It names the controller only while its `bootId` equals `controller_boot_id`. Revocation swaps the boot id, so a revoked lease never keeps it. - `legacyControllerClaim` writes it in the same UPDATE (from `currentControllerProcessStamp()`, no I/O). - `legacy-controller-lease.ts`: `legacyControllerOwnerStamp`, `legacyControllerProvenGone` (calls `proveOwnerGone`), and `settleLegacyControllerLease` (compare-and-swap expiry of the proven controller's outstanding lease after it wakes live-run recovery). - `ownership-registry.ts`: `legacy_controller_lease` registers `settle` and `listForRuns`. The owner is the bound process; other rows are boot-id fences. The listing covers outstanding leases only. - `heartbeat/recovery.ts` (3 small hunks): a controller of another boot that the shared prover proves gone has its live lease settled through `settleOwnableKind` and skips the age guard. The local child-process check after it is unchanged, so a live child still keeps the run running. - Migration 0447 builds the partial index with `CREATE INDEX CONCURRENTLY` (heartbeat_runs is a known-large table) and is added to `MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS`. Bespoke code deleted: none. The kind had no liveness proof; the age guard stays for controllers that are not proven gone. ## Verification - `cd server && npx vitest run src/services/legacy-controller-lease.test.ts`: 33 passed. - `cd server && npx vitest run src/__tests__/ownership-reconciliation-report.test.ts`: 17 passed. - Red with `recovery.ts` and `ownership-registry.ts` from the base commit (tests, schema and lease module from this branch): 4 failed, 29 passed. Red: the crash-point reaper test, the expired-lease age-guard test, the reconciler settle test, the exact-controller CAS test. The tests were written after the code. - New tests: - the claim stamps this process, and a changed boot id unbinds it; - crash point: a controller dispatched the adapter and was killed with its lease live; the startup reaper (default 5-minute age guard) fails the run as `process_lost` at once; - a dead controller's expired lease skips the age guard; - counter-cases keep the run and lease through the reaper and a reconciler pass: a live earlier-boot PID, a dead PID in another PID space, and a previous-release row with no `controller_process`; an expired lease of a live controller stays behind the age guard; - a read-only pass (`record: false`) never settles; a recording pass expires the lease and leaves the run to live-run recovery; - settle refuses a controller that changed after the proof. - Not run locally: typecheck (CI runs it) and the wide suites. ## Risks - Previous-release story (rules/upgrade-safe-run-state.mdc): rows saved by the previous release have no `controller_process`. They stay boot-id fences with today's release (expiry, revocation, age guard). A rollback ignores the column. - Unprovable controllers (another host or PID space, unknown space, a live PID with an unreadable start time) never release early. Their handoff stays the lease expiry plus revocation, and the age guard. A different host is never proof. - Behavior change: a dispatched run whose controller is proven gone is failed as `process_lost` without the 5-minute wait. The retry decision is unchanged. - The settle expires the lease only. Live-run recovery still settles the run under its own checks. Accounting and workspace obligations are not touched. - The index build is concurrent. A cancelled build is retried by the migrator's invalid-index handling. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, local test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 23:27:54 -06:00
DottaandClaude Opus 5.5	0cb9770d1c	feat(recovery): name the owner of teardown claims and login/probe leases (#19965 ) <!-- Write all pull request text in Simplified Technical English (ASD-STE100): short sentences, one instruction per sentence, simple approved vocabulary, and the active voice. --> ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Environment leases carry several in-progress records: login/probe leases and in-flight teardown claims among them > - The ownership reconciler (durable ownership plan, Phase 2) must know who owns each record, or invariant I1 reports a gap > - The ownership data showed one teardown claim and one login/probe lease with no owner stamp > - A teardown claim could also show the wrong owner: a controller stamp left by an earlier attempt named a dead process while a live claimant held the claim > - This pull request names an honest owner for both kinds, without a new write before `turn.start` > - The benefit is that I1 measures real gaps, and the later settle for these kinds can trust the owner it is given ## Linked Issues or Issue Description Refs #19964 Refs #19933 What happened? The ownership reconciler reported `environment_teardown_claim` and `environment_login_probe_lease` records as I1 `no_owner_stamp`. Workspace recovery and native export retry claim teardown without a controller stamp. A login/probe lease never had an owner stamp. A teardown claim that kept an older attempt's controller stamp was attributed to that older process. Expected behavior Each in-progress record names the owner that can still move it: a process proved by identity, or a fence whose expiry is never proof by itself. A stamp for a different attempt never names the owner. Steps to reproduce 1. Let workspace recovery claim a pending-cleanup lease (no controller stamp), or open a login session lease. 2. Run the ownership reconciliation pass. 3. See `no_owner_stamp` for the record, or a dead earlier controller for a claim that a live claimant holds. ## What Changed - `recovery/ownership-registry.ts`, `environment_login_probe_lease`: the owner is the holder process stamped by the write that makes the lease active (`metadata.leaseController`, added in #19964). A lease saved by the previous release still names no owner; it ends with its login session. - `recovery/ownership-registry.ts`, `environment_teardown_claim`: a controller stamp counts only when its `attemptId` equals the claim's `pendingCleanupAttemptId`. Any other claim with an attempt id is a fence: the attempt id is the token and `pendingCleanupLeaseExpiresAtMs` (renewed by the claimant) is the expiry. An expired fence stays unprovable. A claim with no attempt id names no owner. - Report-only. No settle is registered for these kinds in this change. ## Verification - `cd server && npx vitest run src/__tests__/environment-lease-owner-stamps.test.ts src/__tests__/ownership-reconciliation-report.test.ts` (embedded Postgres). - Red on the previous registry, green here (tests written after the code): - login/probe lease is owned by the process whose write made it active - login/probe lease is proven gone when that process has exited - teardown claim never reads a controller stamped for an earlier attempt (fence held by its token) - an unstamped teardown claim whose fence expired stays unprovable, never gone - Green on both: a stamped claim for this attempt with a dead PID is gone; an unstamped login lease and an attempt-less claim report I1. - Full typecheck is left to CI (local memory limit). ## Risks - Low risk. The change only alters the owner the report-only reconciler reads. No record is changed. - Stacked on #19964 (the lease holder stamp). Merge #19964 first. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edit, embedded Postgres tests). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 23:17:40 -06:00
DottaandClaude Opus 5.5	cc072e7c4e	feat(recovery): settle orphaned active environment leases on the shared ownership path (#19964 ) <!-- Write all pull request text in Simplified Technical English (ASD-STE100): short sentences, one instruction per sentence, simple approved vocabulary, and the active voice. --> ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agent runs hold environment leases (local workspaces and remote sandboxes) while they execute > - A crash between the terminal run write and the lease release leaves the lease `active` with no holder > - A sweep released such a lease when the run was terminal and the lease was idle for five minutes. The age was the only guard against a race with the live executor, which releases the lease itself with the run's resource disposition > - Phase 2 of the durable ownership plan moves each in-progress record kind onto one shared owner proof (#19933 added the settle hook) > - This pull request moves `environment_run_lease_active` onto that path: the lease names its holder, the shared prover judges the holder, and the settle never destroys a sandbox > - The benefit is that a dead holder's lease settles on the next pass, a live holder is never raced, and the five-minute age gate is gone ## Linked Issues or Issue Description Refs #19933 What happened? An orphaned active environment lease was released by age: "run terminal plus five minutes" (the reaper in `heartbeat/recovery.ts` and the live-recovery `orphanLease` worker). The ownership data showed one such lease settle 119 s after it was ready. Without the age gate the sweep could flip a lease to `pending_cleanup` (destroy) while the executor or a recovery path was about to retain it. Expected behavior A run lease settles when its holder is proven gone, by identity and not by age. A live holder is never raced. A settle never destroys a sandbox whose files may still be needed. Steps to reproduce 1. Start a run that holds an environment lease. 2. Kill the server after the run status becomes terminal and before the lease release. 3. Restart. The lease stays `active` until the five-minute age passes, then the sweep destroys the sandbox. ## What Changed - `environments.ts`: the write that makes a lease active (insert and same-run reacquire) stamps the holder process in `metadata.leaseController`. The stamp does no I/O (`environment-lease-controller.ts`). - `recovery/ownership-registry.ts`: `environment_run_lease_active` lists leases with their run status. The owner is the run while it is active, then the stamped holder. It registers `settle` and `listForRuns`. - `environment-run-lease-settlement.ts` (new): compare-and-swap settle on the exact row (same run, status `active`, same metadata, terminal run). A sandbox lease gets the stop-and-retain intent (stop and keep, never destroy). Other leases move to `pending_cleanup` as before. It keeps a lease that a pending native export, a legacy shutdown fence, a shared provider resource, or an accepted native result still needs. - `recovery/ownership-invariant-report.ts`: `settleOwnableKind` accepts `currentProcessIdle`. A caller that passes every run this process executes in `skipRunIds` may treat a current-process owner of another run as gone (`current_process_idle`). The periodic pass never does this. - `heartbeat/recovery.ts`: `sweepOrphanedActiveLeases` no longer releases run leases by age. It runs native workspace preservation first (a separate obligation), then calls `settleOwnableKind` with this process's executions. Login and probe leases (no run) keep the idle sweep until their own change. - `heartbeat/live-recovery-work.ts`: a run lease is due now; only run-less leases keep the five-minute idle deadline. - `sandbox-stop-and-retain.ts`: an ownership settle records a recovery marker on the intent and keeps `updatedAt`, so the pending-cleanup sweep stops the sandbox in the same tick. - Deleted: the age gate for run leases and the page-order bump for run leases. ## Verification - `cd server && npx vitest run src/__tests__/environment-run-lease-ownership.test.ts src/__tests__/heartbeat-orphaned-active-lease-sweep.test.ts src/__tests__/ownership-reconciliation-report.test.ts` (embedded Postgres). - Red on unchanged master sources, green on this branch (tests were written after the code): - stamps the lease holder in the write that makes a lease active - settles a lease whose holder died after the run went terminal, on the next pass (crash-point state: terminal run, dead holder in this PID space) - stops and keeps a dead holder's sandbox instead of destroying it - leaves a current-process lease to this process: settled only when it is not executing the run - keeps the run fact alone for one release on a single-host server - names no owner on an ephemeral server (`PAPERCLIP_EPHEMERAL_SERVER=true`) - the three updated sweep tests (sandbox stopped and kept, not destroyed) - Live-owner counter-cases (green): run still active; another live PID; another PID space; current process while executing; read-only pass (`record: false`) never settles; a lease re-stamped after the proof is not settled; a legacy lease whose run this boot owns is kept. - Also run green: `recovery.test.ts`, `heartbeat-observed-workspace-preservation.test.ts`, `workspace-recovery-warm-handoff.test.ts`, `environment-service.test.ts`, `heartbeat-pending-cleanup-sweep.test.ts`, `native-environment-settlement-work.test.ts`, `environment-cleanup-work.test.ts`. - Full typecheck is left to CI (local memory limit). ## Risks - Behavior change: an orphaned sandbox lease is now stopped and kept, not destroyed. Workspace preservation and provider auto-delete decide when it goes away. This can keep stopped sandboxes longer than before. - A recovery path on another process can mark a run terminal while the dead executor's stamp is still on the lease. The settle can then win before that path releases. Because the settle only stops and keeps the sandbox, files are not lost; the later release continues from the stop-only intent. - Previous-release rows: a single-host server keeps the run fact for one release (a run this boot owns is excluded). An ephemeral server never releases an unstamped lease; it stays visible as an I1 gap with no operator action yet. - No migration. The stamp lives in existing lease metadata. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edit, embedded Postgres tests). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 22:43:57 -06:00
DottaandClaude Opus 5.5	bf3a067076	feat(recovery): settle the finalizer and cleanup leases on the shared ownership path (#19950 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs keep their ownership in `native_run_finalizations`, and the shared ownership reconciler now settles proven-gone owners (Phase 2 of the durable ownership plan) > - The finalizer lease (`native-finalizer:`, 5 min) and the retained-session cleanup lease (`native-cleanup:`, 60 s) recorded only a random token. They named no holder process > - A server killed while it finalized or cleaned up kept the run waiting for the whole lease. The lease was released by age alone. A graceful shutdown also keeps an in-progress finalizer lease for the next boot to wait out > - This pull request records the holder process with each claim and moves both kinds onto the shared prover and a compare-and-swap settle > - The benefit is that a dead holder's run continues in the first pass after boot, and a live or unprovable holder is never released ## Linked Issues or Issue Description Refs #19933 (the settle hook and the first moved kind, merged). What happened? A finalizer or cleanup lease whose server process died stayed held until the lease expired (5 min or 60 s). The run stayed `running` and its follow-up work waited. What did you expect? The next boot proves the dead holder gone and frees the lease at once. A live holder, or one that cannot be proven gone, keeps its lease. Steps to reproduce 1. Accept a native result and let the finalizer claim it. 2. Kill the server (SIGKILL) before the finalizer commits. 3. Start the server. The run stays `running` for up to 5 minutes. ## What Changed - `native_run_finalizations.lease_holder` (jsonb, migration 0446): `{token, bootId, pid, hostHash, processStartedAt}` of the process that took a token-only lease. It names the holder only while `token` equals `lease_owner`, so renewals keep it and a later lease never inherits it. The executor's `controller_*` columns are not changed. - The finalizer claim (`native-run-finalizer.ts`) and the cleanup claim (`native-session-executor.ts`) write it in the same UPDATE as the lease, from `currentControllerProcessStamp()` (no I/O). - New `native-token-lease-settlement.ts`: compare-and-swap settle that expires exactly the proven-gone holder's outstanding lease, after it notifies the finalization or cleanup work that the expiry frees. - Both claims also wake the ownership worker queue in their transaction, so a quiet worker watches the new holder (Greptile round 1). - `ownership-registry.ts`: `native_finalizer_lease` and `native_cleanup_lease` register `settle`. Their owner is the holder process; rows without a holder keep the token fence. Listings cover outstanding leases only. - `heartbeat/recovery.ts`: restart recovery settles both kinds before it finalizes and schedules cleanup. A failure is logged and the reconciler pass retries it. - Two sparse partial indexes on `lease_expires_at` for the two listings, so the settle watch re-check stays indexed. Bespoke code deleted: none. These kinds had no liveness check; their only release was lease age (the takeover in the claim), which stays as the fence for unprovable holders. ## Verification - `cd server && npx vitest run src/__tests__/heartbeat-native-restart-suspension.test.ts -t "finalizer\|cleanup\|holder\|exact lease"`: 12 passed (rerun after the rebase on master). The crash-point test also asserts the ownership wake for its run; it fails without the signal. - `cd server && npx vitest run src/__tests__/ownership-reconciliation-report.test.ts`: 17 passed. - Red on the base branch (server code at `bea3bfb20b`, tests and schema from this branch): 7 new tests fail, 5 pass (the 5 are existing tests). Red: the crash-point restart test, the two reconciler settle tests, the three live/unprovable counter-cases (owner was a token fence), and the exact-lease CAS test. The tests were written after the code. - New tests: - crash point: a finalizer claim was written and its holder killed before commit; the next restart finalizes the run, and this boot's claim stamps itself as holder; - a dead finalizer or cleanup holder is settled by a recording pass, never by a read-only pass (`record: false`); the accepted result and phase are untouched; - live counter-cases: a live earlier-boot PID (`pid_alive`), the current boot (`current_process`), a dead PID in another PID space (`different_process_space`, unprovable) keep the lease through a pass and a restart; - a holder written for another token, and a holderless row of the previous release, stay token fences, also with `PAPERCLIP_EPHEMERAL_SERVER=true`; - settle refuses a lease taken over after the proof (`owner_changed_since_proof`). - Not run locally: typecheck (CI runs it) and the wide suites. ## Risks - Previous-release story (rules/upgrade-safe-run-state.mdc): rows saved by the previous release have no `lease_holder`. They stay token fences and the existing claim takes them over at expiry. Neither lease outlives 5 min, so this covers at most 5 min after the upgrade. A rollback ignores the new column. - Unprovable holders (other host or PID space, unknown space): no operator action. The lease is short and self-expiring; at expiry the next claim takes it over and the old holder's next token compare-and-swap fails. That fence is the handoff. A different host is never proof. - The settle only expires a lease. Finalization and cleanup still run under their own checks (cleanup keeps its maintenance-history proofs). Settling ownership does not settle the accepted result, workspace preservation, or accounting. - Migration adds a nullable column and two partial indexes on a sparse predicate. Low risk. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, local test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 22:16:34 -06:00
DottaandClaude Opus 5.5	f6cb3e43f4	fix(admission): admit new work independent of other runs' cleanup and accounting (#19944 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Budget admission decides if new agent work can start in a company > - Five admission gates waited on the cleanup or accounting of other runs > - The restart-chaos campaign found that these gates froze companies for up to 10 minutes, caused false budget pauses, denied in-run writes, and dropped wakes, comments and Retries > - The durable ownership plan (Decision 1) removes the company-wide accounting gate. It does not narrow it again > - This pull request removes the five gates. Unfinished accounting now adds its bounded estimate to exposure > - The benefit is that one run's unsettled accounting never stops the other work in its company ## Linked Issues or Issue Description Refs #19831 (plan, section D), #19838 (inventory, finding 3), #19927 (Phase 2 reconciler data), #19497, #19724, #19826. What happened? Admission refused new work in these cases, with no reached hard stop and no pause: - a terminal run awaited accounting (`waitingRunCount`); - a native run recovered its accounting (`recoveringRunCount`); - a budget-owned scope hold awaited reconciliation, after the hard stop no longer applied; - a budget input was older than 15 s (`budget_evaluation_work`); - the predecessor run row of a task was missing (`hasPendingRunCleanup` failed closed). Expected behavior Admission never waits on the cleanup or accounting of another run. Only a reached hard stop, an explicit pause or an inactive company refuses new work (invariant 3). Steps to reproduce Make a company with an active budget policy. Leave one failed run with `cost_accounting_pending = true`. Request work for another agent. On `master` admission refuses with "Company cannot start work while completed runs await accounting." ## What Changed - `budgets.ts`: admission refuses only for a reached hard stop or an explicit pause. The code removes `observedBlocks`, the `recoveringRunCount` check, the stale `budgetHeld` refusal and the 15 s input-age gate. It also removes the unused `inspectInvocationEligibility`. `reconcileScope` no longer returns accounting holds. - `budget-exposure.ts`: removes the `waiting` and `recovering` aggregates and the own-task subquery. The admission query is smaller. Unfinished accounting stays exposure: the larger of the held reservation and the observed cost. - `budget-evaluation-work.ts`: removes `oldestBudgetInputAt`, which nothing reads now. - `execution-blocker.ts`: `hasPendingRunCleanup` returns false for a missing run row. A run row that is deleted removes its finalization coordinator and nulls its lease references, so the row owns nothing. A predecessor that owns a lease or a coordinator still holds the successor of its own task. - `recovery/ownership-invariant-report.ts`: the I3 check no longer mirrors the removed gate. It reports a wake request or a run that was refused with an accounting-hold reason. Any such report is a regression. - `doc/SPEC-implementation.md`: adds the product rule "Admission is independent of cleanup and accounting". `doc/execution-semantics.md` is aligned. ## Verification - New `server/src/__tests__/admission-independent-of-cleanup.test.ts`, on embedded Postgres. It has one case per gate. All six cases fail on `master` and pass here. A seventh case (an explicit pause still refuses) passes on both. One case also asserts invariant 3 together with exposure: two failed runs that each hold a 400-cent reservation fill a 1000-cent budget, and a hard stop still refuses. - Existing tests that asserted the old gates now assert admission (invariants 3 and 4): `closed-native-receipt-accounting`, `budgets-service`, `budget-evaluation-work`, `heartbeat-held-comment-wake`, `ownership-reconciliation-report`. In `heartbeat-held-comment-wake`, each saved comment, Retry, assignment and continuation is delivered exactly once while another run's accounting is pending. Settling accounting later does not deliver it again. - All six files pass locally, one file per process. The server package typecheck passes. - Speed: admission is on the turn path. This change only removes work from it: two correlated aggregates, one own-task subquery and one `budget_evaluation_work` read per policy. No wait or I/O is added before `turn.start`. - Not run here: the chaos waves (K13 storm, K6 crash loop, K4 long holds). The coordinating session schedules them on the shared harness. ## Risks - Behavior change: work starts while accounting for other runs settles. Spend stays bounded by the reservation capacity check and the hard stop, which read the ledger plus observed cost. A run that has no reservation and no observed cost adds zero exposure until its charge commits. This is the gap that Decision 1 accepts. Budget enforcement was already best effort (SPEC 11.6). - A budget-owned pause with no reached hard stop no longer refuses work in budget admission. The paused company or agent is still refused by the existing pause checks until the budget worker lifts the pause. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 21:46:39 -06:00
DottaandClaude Opus 5.5	c6e4ed1547	feat(recovery): settle a claimed wake whose run finished without settling it (#19934 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A wake request is claimed by the run that handles it, and that run marks it finished. > - In a K4 chaos wave (#19927), native restart recovery blocked a run and ended it, but left its assignment wake `claimed`. Nothing ever released it. > - The durable-ownership plan settles such records on one shared path, by proof, never by age. > - This pull request registers a settle for `wake_request_claimed`. It is Phase 2's second kind and is stacked on the accepted-result lease PR. > - The benefit is that no run-ending path can strand a claimed wake. ## Linked Issues or Issue Description Refs #19831 (plan), #19861 (registry), #19927 (the data that found this). Stacked on #19933 (it adds the `settle` hook). What happened? Run `7c45860b…` failed with `native_restart_recovery_blocked` (`execution_recovery_budget_exhausted`) at 23:19:41. Its wake `6b3c4cce…` stayed `claimed` until the end of the wave (23:44 and later). Rule I2 flagged it. What did you expect? A claimed wake ends when its run ends, with the run's outcome. Steps to reproduce 1. `node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env local --faults K4 --k4-hold-max-seconds 600 --tasks 6 --concurrency 3 --seed 9203 --keep` 2. Look for `agent_wakeup_requests` rows with status `claimed` whose run is terminal. ## What Changed - New `recovery/wake-request-claimed-settlement.ts`: when the owner run is terminal, and for a native run its coordinator is `committed` or `terminal_failure`, the claim gets the run's outcome. That is `completed` for a succeeded run and otherwise the run's status, as run completion and native terminal projection already do. It is compare-and-swap on the claim (id, run, status `claimed`). - `ownership-registry.ts`: `wake_request_claimed` registers that settle. - `native-finalization-reconciler.ts`: the pass log line includes the settled count per kind. ## Verification - `cd server && npx vitest run src/__tests__/ownership-reconciliation-report.test.ts`: 17 pass, including the new case. It covers a blocked native run, a native run whose projection is still pending (kept), a succeeded legacy run, a live run (kept), and a read-only pass (no change). - Red/green (test written after the code): the new test fails on master and passes here. - Server typecheck passes. ## Risks - No persisted shape changes, so there is no previous-release story beyond this: claims saved by an older release are settled the same way. - Race with the run's own settle: it waits for a native coordinator to be final. For a legacy run, both write the same status, so the race is idempotent. - No change adds work before `turn.start`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 21:18:22 -06:00
DottaandClaude Opus 5.5	f5400e2594	feat(recovery): settle the accepted-result lease on the shared ownership path (#19933 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A control-plane restart can kill the controller after an agent's result is accepted. The result then stays leased to a dead process. > - Today a private sweep (`native-dead-controller-result-lease.ts`, #19713) proves that controller dead with its own liveness rules. > - The durable-ownership plan (#19831) replaces private proofs with one prover (`proveOwnerGone`, #19853) and one registry (#19861). > - This pull request moves the accepted-result executor lease to that shared path and deletes the private sweep. It is Phase 2's first kind. > - The benefit is one tested rule for when a dead owner's result lease is released, and the reconciler now repairs a lease that restart recovery missed. ## Linked Issues or Issue Description Refs #19831 (plan), #19853 (prover), #19861 (registry), #19713 (old sweep), #19883 (legacy host fallback), #19927 (Phase 2 data). What happened? The accepted-result lease had its own liveness proof, separate from the shared prover. The reconciler could only report a dead owner, not settle it. When restart recovery missed a lease, it waited out the 20-minute lease. What did you expect? One prover decides, and both restart recovery and the periodic reconciler pass settle a lease whose controller is proven gone. Steps to reproduce 1. Accept a native run result. 2. Kill the controller (SIGKILL) and start a new server. 3. If the restart sweep does not see the row, the result stays leased until the lease expires. ## What Changed - `ownership-registry.ts`: a kind may register `settle` and `onUnprovable`. `native_accepted_result_lease` registers both. Other kinds stay report-only. - New `native-accepted-result-lease-settlement.ts`: a compare-and-swap settle that expires exactly the proven owner's lease. It keeps the old guards: the run must still be `running`, remote copyback stays fenced, and a lease taken over since the proof is left alone. - `ownership-invariant-report.ts`: the pass calls `settle` on `gone` and `onUnprovable` on `unprovable`. A read-only pass (`record: false`) never settles. New `settleOwnableKind()` for restart recovery. - `heartbeat/recovery.ts`: restart recovery calls `settleOwnableKind("native_accepted_result_lease")` instead of the old sweep. - Deleted `native-dead-controller-result-lease.ts`. - `owner-proof.ts`: accepts a board attestation (`operator_attested`). It also keeps the one-release local fallback for rows saved before their PID space was recorded, on single-host servers only, and flags each use so it is logged (`legacy_host_unknown_dead_pid_proof`, #19883). An ephemeral server returns `unprovable` (`process_space_unrecorded`) and the existing "confirm the previous server stopped" board action is raised. ## Verification - `cd server && npx vitest run src/__tests__/heartbeat-native-restart-suspension.test.ts`: 27 pass, 7 of them new. - `npx vitest run src/services/owner-proof.test.ts src/__tests__/ownership-reconciliation-report.test.ts`: pass. - Red/green (tests written after the code): 5 new behaviour tests fail on master and pass here. They cover the ephemeral board action, the reconciler-pass settle, the live-owner counter-case in that pass, the legacy marker, and operator attestation. The stamped-dead-controller and foreign-PID-space cases pass on both, as the old path behaved the same. - Server typecheck passes (`npx tsc --noEmit -p server/tsconfig.json`). - Not run locally: the full server suite (`pnpm verify` asks for broad coverage because a file was deleted). Required CI covers it. ## Risks - Previous-release story (rules/upgrade-safe-run-state.mdc): NRF rows saved before #19868 have no `controller_host_hash`. They keep the same one-release local proof as before on single-host servers, and each use is logged. On ephemeral servers they stay unprovable and ask the board, as before. Rows on another PID space keep their lease (the fence), as before. - Fail-open risk: settle is compare-and-swap on the lease token, boot, PID, host hash and start time of the proven owner. An `unprovable` owner is never released. Live-owner and foreign-space cases are tested. - No change adds work before `turn.start`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 20:58:04 -06:00
DottaandClaude Opus 5.5	988a4b6749	feat(recovery): ownable-kind registry and report-only ownership reconciler (#19861 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A control-plane restart leaves in-progress records owned by dead processes, and about 31 per-subsystem proofs decide when to release them > - The durable ownership plan (section C) asks for one registry of ownable record kinds and one reconciler > - Phase 1 must measure first: how many in-progress records exist, who owns them, and which invariants they break, with no behavior change > - This pull request registers 46 kinds from the Phase 0 inventory's section C and adds a report-only pass to the existing native finalization reconciler > - The benefit is per-kind invariant-violation counts from chaos waves and staging. That data unlocks Phase 2 (moving kinds onto one settle path) and the removal of the company-wide admission gate ## Linked Issues or Issue Description Refs #19831 (plan, section C and Phase 1). Refs #19846 (Phase 0 inventory, section C). Stacked on #19853 (the prover) and #19851 (one process identity). Problem No single place lists the control plane's in-progress records or checks the plan's invariants. Each subsystem releases its own records by its own rules, so nobody can count how many records break "owned", "settled" or "visible next action". Expected behavior One report-only pass lists every registered kind, classifies each owner as gone, alive or unprovable, and records each invariant violation once. It changes no record and stays quiet when nothing is in progress. Impact Phase 2 moves kinds onto the shared settle path in the order these counts show. Phase 3 removes the admission gate once the I3 data exists. ## What Changed - New `server/src/services/recovery/ownership-registry.ts`: 46 `OwnableKind` entries (`id`, `obligation`, `storage`, `listInProgress`, `settle: null`, `invariantChecks`). Each record names its owner as a process stamp, a fence token, a run, an operator, or none. The skill-cache lock is registered with a named coverage gap because its lock files live in per-skill roots. - New `server/src/services/recovery/ownership-invariant-report.ts`: one pass classifies every owner with `proveOwnerGone` (or the owning run's status), then runs: - I1 owned: an in-progress record with no owner; - I2 settled: an owner proven gone on two consecutive passes while the record is still in progress; - I3 admission independence: terminal runs whose accounting outlives the 10 min closure, which hold company admission; - I4 exactly-once intents: wake requests ended `skipped`/`failed` on a temporary refusal (heuristic until section E); - I7 visible next action: an agent-assigned `in_progress` issue with no live run, no pending wake and no open recovery action for 5 min. - I5, I6 and I8 are reported as named coverage gaps; I8 comes in the speed-budget PR. - Violations are written once per process to the activity log as `recovery.invariant_violation` (`actorId: ownership-reconciler`, `details.mode: report_only`), insert-only with no live-event or plugin fan-out. Nothing goes to Telemetry (AGENTS.md §5.7). - `native-finalization-reconciler.ts`: `registerOwnershipReconciliationWork` runs the pass in the shared delivery-work scheduler (new queue `ownership-invariant-report`). There is no second loop. - `index.ts` registers it after residual recovery and does not await its first pass, so readiness is not delayed. ### Sleep-aware trigger (rules/sleep-aware-background-work.mdc) - Trigger: one pass at registration (startup reconstruction from durable truth). After that, a settled `active-execution-recovery` signal, which run lifecycle, controller renewal and process-evidence writes already emit. Signals are coalesced to at most one pass per 30 s. No producer gained a write. - Pending predicate and next deadline: a 30 s follow-up only while the last pass saw a changing in-progress set, or an owner proven gone that needs the second I2 pass. - Quiet: an empty set or the same stuck set twice sets no deadline. The test asserts `nextWakeAt() === null` after an idle pass. - Turn path: the pass never runs before `turn.start`. Stamps read only cached identity. ## Verification - `npx vitest run src/__tests__/ownership-reconciliation-report.test.ts` (embedded Postgres, in `server/`): 4 tests pass: - every kind is registered once with no settle path; - verdicts per class: alive (this process's claim), gone (lease of a failed run), unprovable (no owner; another PID space); - each rule fires on its fixture, and I2 fires only on the second pass; - every touched table is byte-identical before and after (report-only); - activity rows are written once per violation, and none are written with `record: false`; - the worker is quiet when idle and schedules a follow-up only while records change. - `npx vitest run src/__tests__/delivery-work-coordinator.test.ts src/services/owner-proof.test.ts src/__tests__/server-startup-feedback-export.test.ts`: pass. - `npx tsc --noEmit -p server`: no errors. ## Risks - No record changes, so no settle risk. The risk is load: one pass runs about 46 small indexed or `limit 201` queries, at most once per 30 s while runs are active, and none when idle. - I4 matches refusal text with a heuristic. It can over- or under-count until section E gives wake requests closed terminal reasons. It is labelled as such in the code. - Violations dedupe per process, so a restart writes a still-open violation again (once per boot). Chaos reports count distinct `(rule, entityId)`. - Previous-release story (`rules/upgrade-safe-run-state.mdc`): no persisted run state changes. The pass reads rows of any age, and a missing field classifies as unprovable. - `heartbeat_runs` and `native_run_finalizations` record no host or PID space, so foreign-boot owners there classify as unprovable (`missing_identity`). This is the inventory's finding 2, and the report makes it countable. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 13:51:28 -06:00
DottaandClaude Opus 5.5	f439150cb3	fix(recovery): limit the null-host owner fallback to single-host servers and record each use (#19883 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - #19868 and #19871 made restart recovery judge a recorded pid only in the PID space that recorded it > - Rows saved before those host columns existed have no recorded host. They kept the previous local dead-pid proof, so in-flight native runs would not be stranded at upgrade > - On an ephemeral (multi-host) server that fallback is unsafe: the row may have been written on another host > - This pull request limits the fallback to single-host servers, records every use, and gives ephemeral servers a working board action > - The benefit is that single-host upgrades still recover on their own, multi-host servers never accept a cross-host proof, and the next release can remove the fallback with evidence ## Linked Issues or Issue Description Bug description After #19868 and #19871, a `native_run_finalizations` row with `controller_host_hash = null`, or a run with `process_host_hash = null`, still accepted "the pid is dead here" as proof on every server. That includes an ephemeral server, where the row may come from another host. Nothing recorded how often the fallback was used, so it had no safe removal path. Expected behavior - Single-host server: the fallback applies for one release, and each use writes a `legacy_host_unknown_dead_pid_proof` activity-log entry. - Ephemeral server (`PAPERCLIP_EPHEMERAL_SERVER=true`): the row is unprovable, and the board has an action to confirm the previous server stopped. - A mismatched host is always unprovable. Refs #19868, #19871, #19846 ## What Changed - `native-restart-recovery.ts` `evaluateNativeControllerTakeover`: - A coordinator row with `controllerHostHash: null` returns `controller_host_unrecorded` when `ephemeralServerEnabled()` is true. - Otherwise a pid proof from that row carries `legacyHostFallback: true`. - `operator_attested` is accepted as proof (`operator_attested_controller_stopped`). - Receipt-built owners, which have no host key, are unchanged. - New `recordLegacyHostFallbackProof` writes the activity-log entry. It is not a run-log event: run events are progress evidence for restart recovery's crash-loop credit, and the record must not change any recovery decision. It is called from restart recovery, the dead-controller result-lease release, native retirement, and remote warm-session recovery. - New `native-controller-host-recovery.ts`: - `requestNativeControllerStopConfirmation` (board-owned `native_controller_host_unverified` recovery action, idempotent, never displaces another active action). - `confirmNativeControllerStopped`. It verifies the exact recorded controller and stamps `operator_attested` with compare-and-set. It then resolves the action, wakes live-run recovery, and logs `issue.native_controller_stop_confirmed`. - Route `POST /issues/:id/recovery-actions/confirm-controller-stopped`. It is board-only and requires `runtime:confirm_owner_stop`. Generic `restored`/`false_positive` resolution of this cause is refused. A shared `confirmNativeControllerStoppedSchema` validates the body. - UI: `WorkspaceExportRecovery` reuses the "Confirm the previous server stopped" control for this cause, and the generic recovery card hides it. Adds the `issuesApi.confirmControllerStopped` client call. - `recovery/service.ts` orphan reaper: a null `process_host_hash` is judged only when not ephemeral. A successful use records the entry best-effort, after the terminal write. On an ephemeral server the run is left to its existing Stop control. - `ephemeralServerEnabled` moves to a leaf `ephemeral-server-mode.ts` (re-exported by `ephemeral-server-contract.ts`). Importing the contract module from recovery code created an import cycle in the runner child process. - The new route is registered in OpenAPI. - `doc/execution-semantics.md`: documents the host rule and the one-release sunset. ## Verification - `native-restart-recovery.test.ts`: null host on a single-host server is released with the flag. Null host on an ephemeral server is unprovable. A mismatched host is unprovable in both modes. An attested controller is accepted. Receipt owners are unchanged. - New `native-controller-host-recovery.test.ts` (embedded Postgres): - The fallback writes the activity-log entry and no run event. - Ephemeral mode creates exactly one board action. - A wrong run id is refused. - Confirmation stamps the attestation, after which takeover is allowed. The action resolves and the activity is logged. A second confirmation is refused. - `recovery-stale-issue-lock-sweep.test.ts`: on a single-host server a null host is terminalized and records one event. On an ephemeral server it is left alone with no event. The existing audit-failure case still clears the lock. - `WorkspaceExportRecovery.control.test.tsx`: the controller confirmation posts to `confirmControllerStopped` and never to the copyback endpoint. - Locally: these suites plus `remote-warm-session-recovery`, `native-controller-recovery.integration` and `native-startup-cleanup-recovery` (136 passed), and after the CI fixes `native-runner-restart-preparation` and `native-runner-restart-recovery` integration, `openapi-routes` and the suites above (256 passed); the UI recovery suites (52 passed); `node scripts/typecheck-layers.mjs` with no errors in the touched files; `pnpm check:token-gates` clean. ## Risks - Previous-release story: no schema change. Null-host rows saved by earlier releases recover as before on single-host servers, now with an event, and wait for the board on ephemeral servers. The next release removes the fallback; the event count shows whether any rows remain. - On an ephemeral server, a null-host orphaned run process is not reaped automatically. The operator uses Stop. That is the intended "unprovable" outcome. - A board confirmation is an attestation. It applies only to the exact recorded controller (boot id + pid) and only while the host is still unrecorded. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 13:25:47 -06:00
DottaandClaude Opus 5.5	5a20fca9f7	fix(recovery): judge an orphaned run's pid only in the PID space that recorded it (#19871 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The orphan reaper (`terminalizeOrphanedRunningRun`) ends a `running` run whose recorded process is gone > - It judged "gone" with `kill(pid, 0)` on the local host, wherever the process actually ran: another host, another PID namespace, or a sandbox > - So a run whose process is alive elsewhere could be terminalized while it still works > - #19868 fixed the same gap for native coordinator takeover and the runtime-control sweep. This pull request finishes finding 2 for the reaper > - The benefit is that a dead local pid never proves an owner on another host is gone ## Linked Issues or Issue Description Bug description The Phase 0 ownership inventory (finding 2) names the orphan reaper `terminalizeOrphanedRunningRun` (`recovery/service.ts`) as a proof that accepts "the pid is dead here" for a process anywhere. `heartbeat_runs` recorded `process_pid` with no PID space, and a sandbox pid could collide with a local one. Steps to reproduce 1. A run records `process_pid` on host A, or in a sandbox. 2. On host B, that pid number is not alive. 3. B's stale-lock sweep terminalizes the run as `orphaned_running_run`. Expected behavior The process-death authority applies only in the PID space that recorded the pid. Issue-terminal authority is unchanged. Refs #19846, follows #19868 ## What Changed - Migration `0442`: add nullable `heartbeat_runs.process_host_hash`. - `heartbeat/run-lifecycle.ts` `persistHeartbeatRunProcessMetadata`, the one writer of `process_pid`, stores `currentProcessSpaceHash()` for a local pid and `"remote"` for a sandbox pid. - `recovery/service.ts`: the process-death authority runs only when `process_host_hash` is null (rows saved before this column, for one release) or `sameProcessSpace` matches. - Tests: the reaper leaves a dead pid alone for another host, a sandbox, and an unreadable space, and terminalizes it in this space. The writer stamps local and remote pids. ## Verification - `npx vitest run src/__tests__/recovery-stale-issue-lock-sweep.test.ts`: 46 passed. - Red before the fix: with the old `recovery/service.ts`, the three cross-space cases fail (the run is terminalized). Green after. - `pnpm db:generate` produced `0442_low_salo.sql` (one `ADD COLUMN`), and the snapshot set was pruned to five. `pnpm --filter @paperclipai/db check:migrations` passed. Server `tsc --noEmit` shows no errors in the changed files. ## Risks - Previous-release story: runs saved with `process_host_hash = null` keep the previous local proof for this release. That matches #19868, because stranding in-flight runs at upgrade would be a restart failure. New rows are strict. - A run whose process really lived on another host now stays `running` until issue-terminal authority, its controller lease, or another owner settles it. - No new I/O before `turn.start`. The hash is computed at module load, and the stamp rides the existing process-metadata update. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 12:22:15 -06:00
Nicky Leach	45c2bd94c6	fix(native): name the interrupted attempt when an exact resume is refused (#19709 )	2026-11-07 06:21:46 -08:00
DottaandClaude Opus 5.5	e620ce3678	fix(heartbeat): deliver each accepted follow-up exactly once through restarts and admission holds (#19741 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A user can comment, click Retry, or rely on automatic recovery while an agent works on a task > - Restart-chaos waves showed accepted follow-ups being dropped, or delivered twice, around restarts and temporary company admission holds (accounting still settling after a restart) > - A dropped comment means the user's request is never acted on. A duplicate reopens work that is already done > - This pull request makes accepted work wait through a temporary hold and resume when it clears. It also stops a comment from being promoted again after a completed run already received it > - The benefit is that each accepted follow-up is delivered exactly once, eventually, unless the task is cancelled or a reached hard stop applies ## Linked Issues or Issue Description Refs #19702 (never-started tasks during a temporary hold), #19700 (bounded accounting hold), #14910 (stale reopen from a deferred wake; no file overlap). What happened? Restart-chaos waves found these shapes: 1. Comment wake cancelled. A comment was saved behind a run and promoted when the run ended. A temporary company hold refused the unstarted queued run, and claim cancelled it terminally. The recovery wake that followed was cancelled the same way, and the task rested `blocked` without the comment. 2. Dead Retry. A Retry click during the hold stayed `deferred_issue_execution` after the hold cleared. Nothing woke it, and the issue held no execution lock. 3. Refused assignment. A new task's first wake was refused during the hold. Nothing woke stranded-work recovery when the hold cleared, so the task stayed `todo` with no run. 4. Silent comment after recovery. A comment was saved behind a run that restart recovery finalized. A company-scoped hint waited out the 30 s deferred-receipt rescan, so the task looked done with the message unread. 5. Comment delivered twice. A run whose provider process was lost had a saved comment. Its retry started after the comment, received it in its prompt (task history), did the work, and finished the task. The saved wake was then promoted anyway, which reopened the done task. Expected behavior An accepted follow-up (comment, Retry, recovery) is delivered exactly once, eventually, unless the task is cancelled. A temporary refusal never ends in a terminal cancel, skip, or block. Reaching a hard stop remains the board's decision. ## What Changed - `heartbeat/queue.ts`: on a budget block at claim, an unstarted queued run stays queued when the block is temporary. A reached hard stop or blocked unpriced usage still cancels it. - `recovery/service.ts`: stranded-work recovery waits out a temporary hold for any task. It does not escalate to `blocked`. This extends #19702 beyond never-started tasks. - `budget-block-ownership.ts`: a shared `isBoardOwnedBudgetBlock`, moved out of the recovery service. `budgets.ts` is unchanged. - `run-cost-accounting.ts`: settling a run's accounting wakes queued-run dispatch, the deferred-receipt worker, and stranded-work recovery. Held work waits passively; this is what resumes it. - `deferred-execution-notifications.ts`: new `notifyDeferredWakesForIssue`. `native-run-projection.ts` (a run reaching a terminal status) and `environments.ts` (lease release) now hint each saved wake of the issue by id, so delivery does not wait out the company rescan interval. - `wake-queue/adapters/postgres.ts`: release treats a saved comment as delivered only when a succeeded run of the wake's agent on that issue started after the comment's latest revision, and either received the comment as its own wake input (`wakeCommentIds`) or was a retry of the interrupted turn whose task history included it. In that case it cancels the saved wake with "Queued messages were already delivered to a completed run" and leaves the task as it is. Task history alone never consumes a message saved for its own turn. One query checks the whole batch. - `run-cost-accounting.ts`: settling late receipt debt also sends the hold-release notifications. - `doc/execution-semantics.md`: documents the rule. ## Verification Red on `master` and green with this change, on embedded Postgres: - `server/src/__tests__/heartbeat-held-comment-wake.test.ts`: - A promoted comment refused by the hold stays pending, with no cancelled run and no `blocked` task. It is delivered once after the hold clears through real `accountRunCost`. - A Retry clicked during the hold starts exactly once after accounting settles, which wakes its owner. - A new task whose assignment was refused during the hold starts once accounting settles, which wakes stranded-work recovery. - A comment that a later succeeded retry already received as history, or that a later run received as wake input, is consumed without reopening the task. - Guard tests (green on both): - A comment is still delivered in each of these cases: the later run never received it, it was edited after that run started, it was posted while the run was already running, or a non-retry run only saw it as history. - A task whose continuation was refused waits out the hold. - A reached hard stop is still refused. - `server/src/__tests__/native-run-projection-work.test.ts`: projecting a recovered run terminal hints the saved comment's own wake (red on `master`). - Related suites pass locally (14 files, 482 tests): wake-queue, queued comments routes, refused assignment wake, native run projection. - Live replays (local, unmasked, kept instances): - `--replay w-20261107T031005-s7501/t04` (Claude C1, K1): PASS. The comment's run was created at 03:56:24 during the hold, stayed queued, started at 04:06:24 when the hold cleared, and succeeded. It is the only run carrying the comment. - `--replay w-20261107T002911-s7401/t01` (before the per-wake hint): it reproduced shape 4. Run 592d772f was killed 250 ms after `run.result.accepted`, released and finalized at restart, and the task rested `done` with the comment still saved. Calling the existing sweep on the kept database promoted it, which confirmed that the missing piece was the trigger. ## Risks - A queued run can now sit queued for the length of a temporary hold, which #19700 bounds. It does not hold the issue's execution lock, so later comments still coalesce into it. - The delivered-comment rule depends on the run's recorded prompt context. A succeeded run that received the comment in its history counts as delivery, even if the agent did not act on it. This matches exactly-once delivery semantics. - Settling accounting now emits three extra notifications per run. They are coalesced per company. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, run as an agent in Claude Code with tool use (shell, file editing, local test execution, live restart-chaos replays). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 00:24:09 -06:00
DottaandClaude Opus 5.5	08f7ac03dc	fix(recovery): keep a never-started task waiting through a temporary accounting hold (#19702 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Admission checks every wake. A short company accounting hold can refuse new work after a control-plane restart. > - When the hold refused a new task's assignment wake, stranded-work recovery found the task with no run while the hold was still active. It moved the task to `blocked` with a board recovery comment. > - A `blocked` task leaves the stranded-assignment sweep, so nothing woke it after the hold cleared. The task stayed `blocked` with no run, no interruption and no Retry. > - The hold was temporary. Tasks created a few seconds later (after the sweep) started normally once accounting settled. > - This pull request keeps a never-started task `todo` while a temporary budget hold lasts. The existing assigned-todo liveness dispatch then admits it once the hold clears. The task shows the hold it waits for. > - The benefit is that a restart no longer silently drops new tasks created during the accounting settle window. ## Linked Issues or Issue Description What happened? In a restart-chaos storm on master `06bbcea12f`, three new tasks were created while the company accounting hold was active. Each assignment wake got 409 "Company cannot start work while completed runs await accounting." (or "...native run recovers its unfinished accounting."). One task rested `blocked` with 0 runs, interruption `null` and `canRetry=false`. The other two ran later. Root cause `reconcileStrandedAssignedIssues` (`server/src/services/recovery/service.ts`, the `readInvocationBudgetBlock` branch near line 4826) escalates any assigned task to `blocked` when a budget block exists. It does this before the assigned-todo no-run branch, which already skips a budget-blocked task and dispatches it later. So a task that never started was treated as failed recovery work. The refused wake itself is stored as a `skipped` `budget.blocked` diagnostic, which is correct; the escalation removed the only path that re-admits it. Expected behavior A wake refused by a temporary admission hold waits and is admitted once the hold clears, with one run. A real hard stop is still a board decision. Removed assignees and cancelled tasks still do not wake. Steps to reproduce Create a company with a hard-stop budget and a completed run that has `cost_accounting_pending = true`. Assign a new `todo` task. The wake gets 409. Run the stranded-assignment sweep. Before this change the task becomes `blocked` and never runs. Related: #19700 bounds the accounting hold itself. This PR is independent: it handles what happens to a task whose first wake the hold refused. ## What Changed - Stranded-work recovery skips a `todo` task with no run while a temporary budget hold blocks admission. It does not escalate it to `blocked`. The existing assigned-todo liveness dispatch admits the task once the hold clears. - A hard stop (spend at the limit, or blocked unpriced usage) keeps the existing board escalation. Recovery checks this with the read-only `policyBlocks` for the policies of the blocking scope. It does not change budget hold semantics. - The task interruption projection now covers a `todo` task with no run whose latest wake was refused with `budget.blocked`. It shows "Waiting for the budget to allow new work. The task starts automatically when it does." with no Retry. A budget pause shows the existing "budget" user action. - The no-run projection applies the same person-owned gates as the run-backed projection: a budget pause shows the "budget" action, and an ordinary agent, project, company or task pause shows "resume". Only an accounting hold shows the automatic-start text. - `ExecutionBlockerNotice` shows that wait text for a `working` presentation that carries a wait reason. - `doc/execution-semantics.md` records the rule. ## Verification - New embedded-Postgres regression `server/src/__tests__/heartbeat-refused-assignment-wake.test.ts`: - hold active, new task assigned, wake refused (409, `skipped` `budget.blocked`), two recovery sweeps: the task stays `todo` with no run and shows the wait reason. Before the fix this failed with `expected 'blocked' to be 'todo'` (the CHA-9 shape). - hold cleared, two sweeps: exactly one provider turn starts, from `issue.assigned_todo_liveness_dispatch`, and the task finishes. A later sweep does not start another turn. - counter-tests: a terminated assignee and a cancelled task get no run after the hold clears. - Existing `creates a board recovery action for budget-blocked assigned work` (real hard stop) still passes unchanged. - `pnpm exec vitest run src/__tests__/heartbeat-refused-assignment-wake.test.ts src/__tests__/heartbeat-process-recovery.test.ts src/services/task-interruption-presentation.test.ts src/__tests__/heartbeat-issue-liveness-escalation.test.ts` (server) and `src/components/ExecutionBlockerNotice.test.tsx` (ui). - Server typecheck and `pnpm check:token-gates`. - Live restart-chaos storm (local, codex/claude/opencode, 6 in flight, K2 every 3-4 minutes for 20 minutes, no RC-06 mask), revision `e38bb5c8f7`: 17 tasks, 10 pass. - Two new tasks (CHA-16, CHA-17) were created while the company accounting hold was active. Their assignment wakes got 409 "Company cannot start work while completed runs await accounting." Both stayed `todo` (not `blocked`) and showed `state: working`, `canRetry: false`, `retryWait.reason: budget.blocked`, "Waiting for the budget to allow new work. The task starts automatically when it does." No task rested `blocked` with 0 runs. - The hold itself never cleared in this wave (`pendingRunCount: 1` at wave end), so these two tasks did not start before their deadline. That stuck hold is the separate accounting-closure defect that #19700 bounds. The other 5 failures (`R3:retry_no_new_run:B5`) are Retries refused by the same stuck hold. The re-admission after the hold clears is proved by the embedded-Postgres regression above. ## Risks - Low. The change narrows one escalation to tasks that never started, under a budget block that is not a hard stop. Those tasks now stay `todo` and are re-checked by the existing 30-second stranded-assignment sweep, which already re-checks every admission gate. - The hard-stop check reads the blocking scope's active policies once per budget-blocked never-started task in a sweep. It runs only while a hold is active. - The interruption projection adds one indexed read (company + payload issue id) for `todo` tasks in the batch. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 21:12:00 -06:00
DottaandClaude Opus 5.5	4938a930d6	fix(budgets): stop pending or restart-lost accounting from hard-stopping companies (#19497 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Budget admission and run accounting decide whether any task in a company may start > - A control-plane restart that kills a provider turn mid-flight can lose that turn's final usage receipt. ACPX Claude reports usage only at turn end, so the receipt never becomes complete > - Accounting kept such a run "awaiting receipt" forever. Every later admission in the company then refused with "completed runs await accounting", and the budget evaluator opened a hard-stop incident that paused the company. No product action could clear it: raising the budget refuses while runs await accounting, accounting retry finds no new data, and allowing unpriced usage does not apply to pending runs > - Restart chaos waves showed this as a dead task Retry, a new task blocked as "over budget" at 22 of 500 cents, and a hard stop that cancelled queued Retries and an unrelated running task > - This pull request closes the debt from evidence: once native recovery reaches terminal failure and the provider stop is verified, the known part settles as an unpriced charge. Until then the run holds new work as recovering accounting, without a hard stop > - The benefit is that a restart no longer bricks a company with a hard budget. The operator's unpriced-usage policy decides admission, and unknown spend still never shows as zero cost ## Linked Issues or Issue Description Refs #19399, #19426, #19431 (earlier restart Retry fixes; this is the budget layer under the same journey). What happened? After a control-plane SIGKILL during an ACPX Claude turn, the killed run ended `native_provider_terminal_failed` with `usage_json.accountingReceiptReady = false` and `cost_accounting_pending = true`. It stayed that way for good. The task Retry was accepted but parked with `budget.blocked` ("Company cannot start work while completed runs await accounting"), which the UI showed as "Retry is waiting for execution checks". A new Codex task was escalated to `blocked` with "the original recovery target is over budget" while the company had spent 22 of 500 cents. After the next restart the budget evaluator opened a hard-stop incident (`amount_observed` 22.08 < 500), paused the company and cancelled both queued Retries. What did you expect? A restart should not stop the company. Unknown usage from a stopped provider should become an honest, operator-governed state, and Retry should start a run. Steps to reproduce 1. Give a company a hard-stop budget. 2. Start an ACPX Claude task and SIGKILL the control plane mid-turn, so the resumed turn fails. 3. Observe the run stay `cost_accounting_pending` with an incomplete receipt, and every new admission refuse. ## What Changed - `budgets.ts`: completed runs that await accounting now hold new work only. They no longer open a hard-stop incident, pause the company, place an agent budget hold, or deliver Stop to running work. Only spend at the limit, or unpriced usage under a blocking policy, does that. Chaos wave A and a pre-restart lifecycle cell showed the wedge: a run that had just succeeded stayed terminal for about 100 ms before its charge committed, the evaluator paused the company in that gap, and the Stop it delivered left a cancelled run whose receipt never closed. - `run-cost-accounting.ts`: a failed native run is a closed receipt source only with positive stop evidence: its finalization is `terminal_failure` with no result (not a finalization-deadline failure), it no longer records a process identity, and its latest server-authored process event is `native.local_process_stopped`. One SQL predicate serves the reconciler scan and settlement. Accounting now settles its known part as an `unpriced` charge instead of waiting forever. The background reconciler selects these runs. The reservation settles. Other incomplete receipts stay pending as before. A cancelled run keeps its existing accounting drain, so Stop recovery can still deliver its final receipt. - `native-local-process-stop.ts` and its transactional callers: releasing a verified local or remote process stop wakes the accounting worker, so the closed debt settles promptly. - `usage-receipts.ts`: a charge settled without its receipt is marked `accountingReceiptUnavailable`. A later complete receipt that carries its own price (including a rate-card price) corrects it. A receipt with no own price and unchanged usage does not reopen the hold. - `budget-exposure.ts`: the blocking unpriced-usage policy no longer counts a charge settled from a lost receipt with no observed usage. That is unknown, not "usage with no reliable price". Complete receipts that report no tokens (for example Cursor Cloud) still count. A live wave under the default policy hard-stopped its company at 0 of 500 cents on one such charge. - `budget-exposure.ts`: a terminal-failure native run with an incomplete, not capture-failed receipt counts as recovering accounting. Admission holds new work ("New work must wait while a native run recovers its unfinished accounting") without a hard-stop incident, a company pause or cancellation of running work. - `recovery/service.ts`: the stranded-work comment names the actual budget gate instead of always saying "over budget". - `task-interruption-presentation.ts`: a Retry parked by `budget.blocked` says it waits for the budget, not for "execution checks". - `doc/SPEC-implementation.md`: documents the closed-receipt contract next to the existing incomplete-receipt rule. ## Verification - New embedded-Postgres suite `server/src/__tests__/closed-native-receipt-accounting.test.ts`, built from the persisted shape of the killed run: - Red before the fix: the run stayed pending, `reconcileRunCosts` accounted 0, and admission refused with "completed runs await accounting". - Debt stays pending with no verified stop event, with a finalization-deadline failure, with a recorded process identity, and while the coordinator can still produce a result. - Under a blocking policy, a lost receipt with no usage does not hard-stop; one with tokens still blocks. A run with a still-recorded process identity paused the company with a hard-stop incident. - Green after: the stopped run settles as one `unpriced` charge with the reservation settled; admission is clear under an `allow` policy and names "no reliable price" under `block`; a run with a recorded process identity or a non-terminal coordinator stays pending; the closing window holds admission without an incident or pause; a final receipt that arrives later still enters late-receipt reconciliation. - Pending accounting from a run that just finished holds admission with no incident, no pause, no enforcement version bump, and an unrelated running run stays running. Red before the `budgets.ts` change: the company was paused. - A later priced receipt corrects the unavailable charge to its price; an unpriced one leaves it settled. - `task-interruption-presentation.test.ts`: the `budget.blocked` Retry wait message. - `budget-enforcement-scoping.test.ts`: the accounting-hold release cases now assert that pending accounting places no budget hold and that admission clears after settlement. - Existing suites pass locally: all budget suites (control publication, enforcement scoping, enforcement worker, evaluation work, live attempts, history, service, live usage, budget cancellation), native-stop-accounting, explicit-native-continuation, native-case-handoff, cost-data-quality, heartbeat-cost-accounting, run-cost-accounting, native-usage-delivery. Server typecheck passes. - Runner e2e cell `lifecycle-baseline.runner-codex.local.lifecycle-dependency-restart-neutral`, which failed on master with a false agent budget hold before any restart, passes on this branch. - Live restart-chaos wave (local runner, K1 SIGKILL, codex + 2× ACPX Claude + OpenCode, 4 restarts, company policy `unpricedUsagePolicy: allow`) with the accounting fix: every killed run settled (no `cost_accounting_pending` left), the company stayed active, the OpenCode Retry completed the task on the first click and the Codex Retry finished its task. That wave also exposed a 1-second false hard stop in the closing window (observed 0 of 500 cents) that cancelled the running OpenCode turn; the `budget-exposure.ts` change fixes it and has its own red/green test. - Confirming live wave with both commits, same plan (2× ACPX Claude, OpenCode, Codex; 4 K1 restarts): both Claude tasks passed R3 (one Retry click started a run that completed the task), OpenCode passed R1 (continued by itself), and the Codex Retry completed its task. Zero budget incidents, zero cancelled runs, zero runs left pending accounting, company active; three killed runs recorded as `unpriced` charges. The only open signature is an unrelated workspace-durability check (`R2:pre_fault_files_lost` on the Codex long-bench task). ## Risks - Behavior change: a stopped native run's lost receipt now becomes an `unpriced` ledger charge with its known usage. If that usage has tokens, the default `block` policy still stops new work with "recorded usage has no reliable price". If it has none, the charge is visible in Costs but does not stop the company. Other unpriced charges keep their existing effect. Unknown spend is never recorded as a priced receipt. Budget enforcement stays best effort, as the spec says; killed OpenCode and Codex requests already lose their in-flight usage. - Remote runs and runs without a verified local stop keep their debt pending (an admission hold only, no hard stop). - Contract change: pending accounting no longer pauses a scope or stops running work. New work still waits with "completed runs await accounting". A scope's real hard stop (spend at the limit, or blocked unpriced usage) is unchanged. - If a provider stop can never be verified, the run stays "recovering accounting": new work waits without a hard-stop incident. Before, the same state paused the company. The hold is visible in admission errors and Costs. - A receipt that arrives after settlement posts a late correction. The original unpriced charge stays in the ledger. - No schema or migration change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file editing, embedded Postgres tests, live restart-chaos harness). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 14:27:51 -06:00
DottaandPaperclip	124b2cd0ad	fix(server): certify background deadlines for idle tenants (#19001 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Background services own future checks, retries, and recovery work. > - An idle tenant can stop only when a durable wake covers its remaining duties. > - The current alarm catalog covers several domains but leaves these owners blocked. > - A saved future timestamp alone cannot prove that reconstruction or admitted work has settled. > - This pull request binds durable domain snapshots to the actual registered worker revisions. > - The tenant can hand off supported future duties and retain all checks for uncertain or active work. ## Linked Issues or Issue Description No matching public issue was found. Related work: [domain deadline handoff #18656](https://github.com/paperclipai/paperclip/pull/18656), [owned blocker diagnostics #18632](https://github.com/paperclipai/paperclip/pull/18632), and [completed history proof #18646](https://github.com/paperclipai/paperclip/pull/18646). The separate [native idle-session change #18988](https://github.com/paperclipai/paperclip/pull/18988) covers runtime retirement. [Far-future retry re-probes #13534](https://github.com/paperclipai/paperclip/pull/13534) changes recovery responsiveness after an upstream success; this change preserves existing retry decisions. What happened? Future status-card, external-object, skill-sync, watchdog, retry, and quota duties lacked complete owner-bound coverage in the idle deadline proof. A completed quota task could also retain its monitor policy and timestamp and remain blocked after the scheduler had disarmed it. Expected behavior A settled registered owner can hand off its exact supported future deadline. Active work, dirty reconstruction, transaction uncertainty, changed revisions, claims, unknown state, and unsupported duties keep the tenant awake. Completed history retains its data and stops recurring work when the owner has no remaining duty. Steps to reproduce 1. Produce a supported future duty through its normal service. 2. Let the registered worker finish reconstruction and admitted work. 3. Request an owned idle report with the negotiated domain deadline catalog. 4. Check the saved earliest deadline and generation before durable alarm registration. 5. Stop and reconstruct the worker. Check that ordinary admission and claims resume useful work once due. 6. Complete a quota task. Keep its saved policy and timestamp. Check that linked unfinished work still blocks sleep and that settled history does not. Paperclip version or commit The owned patch was qualified at `699c8569fa34338dd75cdc5731381ec1e2efdb45`. Publication head `491fac368ee72eaac8ae105d53e6bf6b305d8a6a` normally merges actual master `039e0dad2c51c80251e1302a19c5566e039c357a`. All 43 owned files are byte-identical. All 29 new upstream paths are preserved. The later normal current-master integration is `8c912e1ba4fc66397a7341ade59373af8075068f`, described separately below. Deployment mode Source checkout with real PostgreSQL 17 and a controlled local Node process. No deployed alarm, tenant suspension, database suspension, or fleet result is claimed. ## What Changed - Add registered-worker reconciliation revisions and invalidate them on stop. - Keep status-card, external-object, and skill deadline interpretation with their owners. - Reuse the watchdog classifier against the supplied database snapshot. - Validate the exact saved retry run and paired queued wake. - Validate legacy quota source bindings and the current assignee or reviewer policy. - Preserve common issue scope, dependency, sibling, and state checks for covered waits. - Compose normalized owner snapshots within the existing bounded deadline proof and recheck revisions after inspection. - Bind actual service and worker instances from server startup. Missing bindings preserve existing blockers. - Retain completed quota policy and timestamps while independently rejecting linked unfinished runs or wakes. - Block unconsumed budget evaluation inputs through the owning queue predicate. Accounting policy and settlement stay unchanged. - Bind completed quota history through fixed-size UUID-array parameters, preserve exact company/task/source associations, and guard external references before the run-ID join. - Batch watchdog subtrees and shared classifier facts without changing the 64-query or two-second proof budget; preserve exact company pairing for first-run evidence. - Extend production-path regression controls and document supported coverage and limits. ## Verification Current integration head: `688abc6cac6016b9fb50e5832f81e2975ed5d49f`. Fresh exact-head checks completed with 58 successes, three intentional skips and one neutral review, including aggregate `ci / verify`. The complete current Greptile review is 5/5. The complete current Ponytail verdict is Ship (4/5), with only optional cleanup of an unused MCP proof input; both prior correctness findings are resolved. All review threads and nested pages were inspected. The normal master merges preserve the registered retry deadline owner and current startup completion log after the same readiness await. The earlier focused evidence below remains tied to its tested source revisions. Current-master integration `8c912e1ba4fc66397a7341ade59373af8075068f`: - Normally merge actual master `46fb933b0155d183ce64a8bdc4950d1b58c926f6`. Every nonoverlapping tree blob retains its owning parent across 14,044 paths. All 42 owner-only files and 15 critical binding/proof/readiness files remain byte-identical. Three automatic overlaps preserve the existing watchdog binding, conservative unavailable-reader fallback and optional actual owner arguments. The two scoped-deadline conflicts retain the entire upstream resource-reservation/mid-pass admission/abort algorithm inside the existing reconciliation revision/running/failure fence, with both parent test sets. - Actual TypeScript 7.0.2 server noEmit passed in 9.434 seconds, normal source declaration layers in 12.102 seconds, and four selected test/helper/config layers extending the actual CI configuration in 3.167 seconds. No compiler option weakening, heap override, donor declarations or SDK replacement stub. - Focused PG0 controls passed: 102 scheduler/proof/owner controls in 16.451 seconds, including all 21 scoped-deadline cases; 121 startup controls; and all 107 unchanged instance-settings route cases in 6.456 seconds. The latter first hit a local prewire-guard listener-registration race. Only the ignored guard was corrected to register the same actual OS-bound server port immediately after native listen returns. Closed, unknown, foreign, offhost and PG destinations remain blocked. Original failure evidence is retained; product code and assertions were unchanged. - The exact current UI policy command now reports zero findings (0.362 seconds); its seven scanner controls pass. The earlier dependent PR used an old scanner/source baseline under a newly enforcing trusted workflow and reported 397 findings. Carrying the genuine current master resolves that mismatch without editing unrelated UI sites or relaxing the gate. - Every meaningful worker checked its executing PID/thread/preload hash and matching receipt. PG0 controls opened no PostgreSQL sockets; all owned processes exited. Original dependency-cache/DNS, ignored-harness syntax/type and guard failures remain separate from product results. - This is fresh semantic and focused runtime qualification, not a local whole-repository or current-head PostgreSQL acceptance rerun. The upstream workspace-busy parked-retry/terminal-holder notification path remains unchanged by this merge and needs its separate current-graph producer check. Historical six-case/scale results below stay tied to their measured sources. A matching controlled default-cadence follow-up is in #19191; no fresh current-head two-cycle or deployed suspension claim follows from this merge. - All 59 exact-head check contexts now pass (55 success, one neutral, three intentional skips), plus the legacy security status. The complete edited Greptile review is 5/5 for this exact commit, with all root/nested pages inspected and no unresolved threads. Ponytail stopped without completing its new turn; it supplied no new verdict or finding. Prior actual findings remain addressed. Earlier scores/checks below belong to their explicit earlier commits. On frozen clean merged source `699c8569fa`: - Real PostgreSQL acceptance: 6/6 passed,83.483 seconds wall. Actual retry producer creates the exact saved run/wake; stopped and reconstructed owner resumes through current admission, one durable start, real local process output and normal task completion. Includes cancellation, drain arrival, retained quota history with queued-run/wake negatives, watchdog review settlement and budget-input empty/pending/consumed control using the actual owner. - Affected coordinator, deadline, scoped-deadline, recovery and composed-proof controls: 78/78 in five files passed,32.114 seconds wall, worker 1/no file parallelism. - `pnpm exec tsc --noEmit -p server/tsconfig.json`: passed,17.467 seconds. Native TypeScript 7.0.2 and Node 24.22.2. - PostgreSQL tests use one dedicated test database, matching fixture/admin URLs, pool limit four, one worker, and no embedded fallback. Every run recorded unchanged source hashes and a clean checkout. - The eight-second real-clock retry is a domain execution check. It is below the Cloud minimum wake lead and is not a positive accepted hosted sleep cycle. - The completed quota-history product failure and original fixture/protocol failures were preserved; the owning-reader fix passed a real PostgreSQL red/green regression before the fresh merged six-case pass. - After runtime completion, the dedicated database had zero other clients and no owned runtime processes. All three qualification process PIDs exited. The coordinator independently audits and drops the database. - Comment-only cleanup has equal strict TypeScript parser ASTs across 15 files. The final lexical audit finds no added comments relative to actual master. `git diff --check` passed. - Earlier retry fixtures omitted responsible-user membership, advanced JavaScript time past the real PostgreSQL lease clock, or produced stdout without normal task completion. Corrected fixtures use real membership, an eight-second real deadline, and ordinary task comment/completion. The last case exposed legitimate disposition repair rather than duplicate retry promotion. - The quota and watchdog controls first ran before admitted work or the existing 30-second reconstruction duty settled. The stronger tests require an interim blocked report and then exact settled-owner proof. Production deadlines were not changed. The separate completed-quota-history defect was fixed in its reader and verified with linked unfinished-run and wake controls. - The status/object and skill owners reviewed the actual final startup, route and post-backup bindings. Their domain source evidence carries where token-equivalent. The master coordinator coalescing change was reviewed separately. Historical domain PostgreSQL results are not presented as fresh full-suite results on this head. The later master merge adds retry predecessor cleanup checks under the existing issue claim lock. The source audit confirms unchanged saved deadlines and run/wake identities. The three actual retry lifecycle/cancellation/drain cases passed on the batched candidate with this parent (31.817 seconds). Those local dependency controls preceded the current exact-head remote run. Native backup preservation and TestOps changes from master are retained. No pass is inferred for changed dependency behavior solely from unchanged owned files. Review repair published as `643f1e7400`: - The original reader returned `unknown` after seven actual watchdog reviews settled across seven companies. The unchanged composed-report regression passed after batching; the final reader passed in 61.841 seconds, including active queued-work and stopped-owner rejection. - A separate two-company red/green control rejects terminal-run context from another company as first-run grace evidence. Shared facts are serialized once, with per-root IDs and review summaries. - The affected recovery, proof and diagnostic unit controls passed 47/47 (14.575 seconds). Startup owner-contract fixtures passed 121/121 (10.589 seconds). - Diagnostic PostgreSQL controls passed7/7 (12.406 seconds), including the normal budget-input producer and its exact blocker reason. Initial empty-state setup uses the same cleanup as per-case teardown when a dedicated test database is reused. - Actual repository `pnpm exec tsc --noEmit -p server/tsconfig.json` passed 12.710 seconds with no heap override. Final source has no added code comments; whitespace checks pass. - The original query-budget failure, two-company discriminator, initial cross-suite fixture failure and an interrupted incorrect local-driver selector are retained separately. The interrupted run is not a pass. Independent cleanup found zero owned clients or runtime processes. Quota-scale repair published as `6696a37f8a`: - The exact 10,000-row completed-history case reproduced `MAX_PARAMETERS_EXCEEDED`:90,002 bound parameters exceeded the driver's 65,534 limit, and the composed report returned `unknown` despite settled owners. Original failure and actual driver diagnostics are retained. - Fixed pending-work queries use three typed UUID-array parameters plus the result limit, keeping exact company/issue/source tuple bindings. The primary run-ID column stays uncast; a guarded CASE validates the raw external reference before conversion. The 10,000-row cap and 64-query/two-second proof budget are unchanged. - The same 10,000-row composed-report case passed 18.860 seconds, with unfinished run/wake rejection, malformed-reference safety and conservative 10,001-row overflow. Its fixture derives retained history from the actual quota producer and normal task completion, expands valid persisted issue/source pairs, then starts the actual registered owners. - The existing actual quota producer case passed 14.187 seconds, including exact future deadline, retained completion history, unfinished work, real foreign-company source references and noncanonical/malformed references. - Affected owner/proof controls passed 22/22 (26.444 seconds). Production server noEmit passed 16.039 seconds. - CI on 643 identified TS7034/TS7005 in the settled-watchdog fixture's initially untyped collection. It is now explicitly typed. A separate affected-entry semantic check extending the repository's actual `server/src/__tests__/tsconfig.json` passed 15.093 seconds: six affected tests, two owner files, ambient declarations and current imported source, with unchanged strict/module/lib/path settings. The production server config excludes that test directory and is not presented as test-layer proof. No custom suppression or replacement SDK stub was added. - All runs were sequential with pool max 4, worker 1, source/config/command hashes and cleared inherited provider/LIVE/other-database variables. Cleanup found zero owned database clients or local runtime processes. At prior head `6696a37f8a5718cd0f79054969a0084462d41d47`, all required CI gates pass. The exact-head inventory has 55 successes, one neutral review and three intentional skips. Greptile is 5/5. The complete latest Ponytail verdict confirms the quota finding is fixed; its only remaining note is optional unused MCP proof state, not a behavior defect. All three review threads are resolved. The earlier 643 result remains historical. The repository-wide typecheck, tests, build and browser gates passed in the prior `6696a37f8a5718cd0f79054969a0084462d41d47` CI run. The new `8c912e1ba4fc66397a7341ade59373af8075068f` full CI also passes. They were not run as a full local workspace command in the focused PostgreSQL lease. Local evidence above uses bounded external databases and does not claim embedded-cluster, live-provider or deployed-alarm qualification. ## Risks - An unsupported or uncertain owner state must remain blocked. An optimistic exemption could lose work while compute is stopped. - Changes during reconstruction or inspection must invalidate the proof. A memory-only timer is not durable alarm coverage. - Current authorization, budgets, cancellation, accounting, and resource ownership still apply on dispatch. The proof does not grant execution authority. - The supported writer model uses in-process transaction signals and startup/resume reconstruction. Independent SQL writers and other replicas need an explicit wake contract. - Native, review, coalesced, and taskless retry variants remain conservative until their owner cases qualify. Reviewer quota and hosted alarm qualification remain open. - The shared catalog remains opt-in. Legacy callers and missing owner bindings retain their old blockers. ## Model Used OpenAI `gpt-6.1-sol` was the configured model for the delegated service implementations and reviews. OpenAI Codex, based on GPT-6, performed managing review and integration. Both used repository tools and code execution. The exact served backend snapshots and context-window limits were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id - [x] I have run tests locally and they pass on the final merged head - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green — exact current-head run passes - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups — complete current-head review and all paginated threads inspected - [ ] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-06 13:48:55 -06:00
DottaandClaude Opus 5.5	a4bb059f59	fix(heartbeat): retry transient sandbox provider lease refusals and explain the block (#19412 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A task runs in an execution environment. A managed sandbox environment gets a lease from a sandbox provider (for example Daytona) before the agent starts. > - When the provider answered sandbox create with a temporary error (HTTP 502), the run failed as the generic `setup_failed`. > - `setup_failed` is the fail-closed code for unknown setup errors. It gets no automatic retry, so the task went to `blocked` at once. The task showed only "Couldn't finish this attempt". > - A user had to find and press Retry. The Retry then worked, which shows that the failure was temporary. > - This pull request classifies temporary provider refusals during lease acquisition from structured fields. It retries them within the existing bounded bootstrap budget, and it explains the block when the retries are used up. > - The benefit is that a short provider outage no longer stops a task. When the outage continues, the task shows the cause and a Retry that works. ## Linked Issues or Issue Description What happened? A task was assigned to an agent that uses a managed Daytona sandbox environment. Sandbox creation returned `HTTP 502`, and the cleanup of the provisional sandbox could not be confirmed. The run failed with `setup_failed` and the error `Failed to acquire lease for environment "Paperclip Sandbox" (sandbox): Daytona sandbox creation failed (HTTP 502); cleanup could not be confirmed for paperclip-create-…`. Three seconds later the task was `blocked`. No automatic retry was scheduled. The task showed no specific reason. A manual Retry started a new run that completed normally. Expected behavior A temporary provider error (HTTP 5xx, 429, timeout) before the agent starts must retry automatically with bounded backoff. The task must show why it waits. If the retries are used up, the task must show the cause, a next action, and a Retry that works on the first click. Steps to reproduce 1. Configure an agent with a sandbox environment. 2. Make the provider return HTTP 502 on sandbox create. 3. Assign a task to the agent. 4. See that the run fails with `setup_failed` and the task goes to `blocked` with no retry. Paperclip version or commit `master` at `612da1e8e0`. ## What Changed - Plugin SDK: an `environmentAcquireLease` error now carries a closed diagnostic envelope (HTTP status or a closed errno code) across the worker RPC. It takes the status of the create call only, not of the compensating cleanup lookup. The existing creation-cleanup ownership envelope is unchanged. New module: `environment-acquire-error.ts`. The resume diagnostic helpers are shared. - Server: new `EnvironmentProviderUnavailableError` and code `environment_provider_unavailable`. The environment run orchestrator sets it as the cause of `lease_acquire_failed` only when structured evidence shows a transient status (408, 425, 429, 500, 502, 503, 504, or a closed connection errno). It never parses message text. Saved-sandbox resume and stopped-worker refusals keep their own codes. - Setup failure (`failRunSetup`): a fresh, pre-dispatch run records `environment_provider_unavailable` with a `providerFailure` (phase `startup`, retryability `transient`). The run uses the bounded bootstrap retry (30 s, then 60 s). An accepted interaction continuation keeps its own continuation retry, as for `plugin_worker_unavailable`. - Recovery: the new code is registered with the other bootstrap codes. It is not sent to the generic continuation retry. It does not create an execution-reconciliation hold, because no provider work started. On exhaustion the stranded notice has the title "Sandbox provider was temporarily unavailable" and a specific next action. - UI: `ProviderFailureNotice` labels the `sandbox` provider as "Sandbox provider". The task notice shows the provider failure, "Paperclip will retry automatically" while a retry is scheduled, and Retry after exhaustion. - Docs: `doc/execution-semantics.md` describes the policy and the sandbox ownership rule. ## Verification Red before the fix, green after the fix: - `server/src/__tests__/heartbeat-process-recovery.test.ts` › "transient sandbox provider refusal during lease acquisition": - "retries automatically and runs exactly one successful successor without user action": the real heartbeat. The lease runtime returns a Daytona 502 once, then delegates. Before the fix: one `failed` run and no successor. After the fix: one `scheduled_retry` successor with no hold and a `retry_scheduled` presentation. After promotion, exactly one successor succeeds. - "explains the block with a working Retry once automatic retries are used up": three refusals. The task is `blocked` with a "Sandbox provider was temporarily unavailable" notice. The presentation is `{ state: "failed", canRetry: true, providerFailure: { httpStatus: 502 } }`, and no execution-reconciliation hold exists. One `requestTaskRetry` is accepted, and its run succeeds. - `server/src/services/heartbeat/run-completion.test.ts` › "routes a … sandbox provider lease refusal through bounded bootstrap recovery": the real orchestrator wrapping. Only the fresh, structured 502 is retried. A message-only 502, a 400, a run after provider dispatch, and a retained native run stay `setup_failed`. - `packages/plugins/sdk/tests/environment-acquire-error.test.ts`: the real worker RPC. The test checks the create status (not the 404 cleanup lookup), the errno, no provider text, the unchanged cleanup envelope, and that forged envelopes are rejected. Commands: ```sh pnpm --filter @paperclipai/plugin-sdk exec vitest run tests/environment-acquire-error.test.ts tests/environment-creation-cleanup.test.ts tests/environment-resume-error.test.ts cd server && npx vitest run src/__tests__/heartbeat-process-recovery.test.ts src/services/recovery/ src/services/heartbeat/run-completion.test.ts src/services/heartbeat/retries.test.ts src/services/legacy-execution-recovery.test.ts src/services/task-interruption-presentation.test.ts src/services/task-retry.test.ts src/__tests__/environment-runtime.test.ts src/__tests__/heartbeat-retry-scheduling.test.ts cd packages/plugins/sandbox-providers/daytona && npx vitest run pnpm --filter @paperclipai/plugin-sdk typecheck && pnpm --filter @paperclipai/server typecheck ``` I did not run a live Daytona 502 on staging. Re-check on staging after deploy (see Risks). ## Risks - Sandbox leaks or double creation: each retry is a new run with a new durable creation record and a new provider name (`paperclip-create-<operationId>`). The failed attempt stays owned by its own record, and the existing creation reconciler settles it. A retry never reuses or abandons it. While a provisional sandbox waits for reconciliation, up to two more sandboxes can be created for the same task. This is the same as a manual Retry today. - Plugins built with an older SDK do not send the envelope. Their refusals stay `setup_failed`, which is the current behavior. - An in-process error with `status`/`statusCode` 500–504 or a connection errno during lease acquisition is now retried twice. The budget is bounded and shared with the other bootstrap codes. - An RPC timeout on the host side is not classified by this change. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through Claude Code as an agent with tool use (shell, file edits, test runs). Extended reasoning was on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 00:08:44 -06:00
DottaandClaude Opus 5.5	6ca7b151f7	feat: compact an agent's context from the task Properties panel (#19378 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A task's Properties panel shows the assignee agent's context window (paperclipai/paperclip#19273) > - The board can see that the context is nearly full, but it cannot act on it > - A full context makes the agent compact at an unplanned time, often in the middle of work > - This pull request adds "Compact now" to the Context hover card > - The benefit is that the board can compact the agent's context between runs, at a time it chooses ## Linked Issues or Issue Description Refs paperclipai/paperclip#19273 Problem The Context row shows how full the agent's context window is. The board cannot ask the agent to compact it. Proposed solution A "Compact now" button in the Context hover card. It runs the harness's own compaction: Codex `thread/compact/start`, and Claude's `/compact` command through ACP. Alternatives considered A separate PRP `session.compact` command. Both harnesses run compaction as a turn, so a compaction turn keeps the existing durable turn binding, interruption, and recovery. A separate command needs a second turn lifecycle. ## What Changed - UI: the Context hover card shows Compact now with three states: available, Compacting…, and unavailable with a reason (agent mid-run, harness unsupported, task closed). The Storybook story shows each state. - Server: `POST /api/issues/:id/context/compact` (board only, `decideIssueAccess` `issue:mutate`, activity log `issue.context_compaction_requested`). It queues a short run marked by `contextSnapshot.contextCompactionRequestId`. `GET /issues/:id/context-usage` returns the `compaction` state. - Server: a compaction run does no task work. The native status arbiter preserves the task status. The lifecycle skips handoff, liveness continuation, review-path disposition and the issue comment, as it does for goal-control runs. - Runner TS: `startTurn({ compact: true })`. The Codex harness session sends `thread/compact/start`. The runnerd transport maps it to PRP `turn.start` with `compact: true` (Codex) or `text: "/compact"` (Claude/ACPX). The native runtime option `sessionCompaction` requires a resumed session and settles with a synthesized yielded result. - Runner Rust: `turn.start` accepts `compact: true` for Codex. `CodexProvider::start_compaction_turn` sends `thread/compact/start` and binds the turn from the `turn/started` that follows the empty response. - Fix (from #19273): Codex emitted `context.compacted` on both the start and the end of one compaction item. It now emits it once, and the count is distinct by `compactionId`. - Fix (from #19273): the conversation event view dropped `context.compacted`, so a finished run never showed the divider. The view now keeps compactions, plus context readings for runs that compacted. A finished run's compactions show as thread dividers. - Fix: the Codex harness stamps before/after sizes on the compaction event. Codex reports the compacted size before the compaction item ends, so the old neighbour-based guess showed the wrong size. ## Verification - `pnpm -r typecheck`, `pnpm build`, `pnpm check:token-gates`, `node scripts/storybook-catalog.mjs`: pass. - New tests: `server/src/__tests__/context-compaction.test.ts`, `status-arbiter.test.ts`, `issue-context-usage.test.ts` (duplicate compaction counts once), `ui/.../context-usage-row.test.tsx`, `native-session-runtime.test.ts` (context compaction), Codex driver and runnerd transport tests, Rust `codex_compaction_turn_binds_the_turn_reported_after_its_empty_response`. Rust `codex_provider` suite: 115 passed. - `pnpm test:run` locally: one failure in `chat-channels.integration.test.ts`. It reads `.claude/skills/slack/SKILL.md` from the main checkout, which is not present in this worktree. It is not related to this change. - Live, local worktree instance, Codex agent on the Paperclip runner: clicked Compact now. The run succeeded. Context went from 35k to 9.4k tokens, and later 9.4k to 9.2k. The task stayed In Review. The thread shows "Context compacted · 9.4k → 9.2k tokens" and the meter dropped. - Live Claude proof: pending. The local Claude agent needs an Anthropic API key. claude-agent-acp handles `/compact` as an echo-less turn, and the runner already maps its "Compacting completed." notice to `context.compacted`. ## Risks - A compaction run resumes the agent's session. If the session cannot resume, the run fails with `native_context_compaction_unavailable` and the task is unchanged. - Compact now is not available while the agent runs on the task, because compaction interrupts the active turn. - The conversation view now returns context readings for runs that compacted. Runs that never compacted return the same events as before. - An old runner binary ignores `compact` and rejects the `turn.start` because it has no text. It does not start an ordinary turn. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 22:07:12 -06:00
Nicky LeachandClaude Opus 5.5	f9d366acdf	fix(runner): require a managed AI connection for plugin Runner-task environments (#19341 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A plugin can provide Runner execution through an environment driver that declares `supportsTasks: true`. Agent computers use this path. > - Core let an agent without a managed AI connection run on that path. The run then used the account that was installed on the plugin's execution host. > - That is not a supported credential source. When the host Runner did not match, the run failed three times, 30 seconds apart, and then blocked the task with a generic error. The error did not name the fix. > - The plugin cannot refuse these runs itself. `describe` has no connection data, and Core replaces plugin error text before the user sees it. > - This pull request makes a managed AI connection a requirement for every plugin Runner-task environment. Core refuses a run without one before it acquires a lease or calls the plugin. > - The benefit is one clear notice that tells the operator to choose an AI connection, and no wasted retries or plugin work. ## Linked Issues or Issue Description Refs #19326. That pull request makes a Runner revision mismatch retry. It noted that runs without a managed AI connection reach the Runner check on a different path, which still blocks after 3 attempts. This pull request removes that path for plugin Runner-task environments. Problem or motivation A `paperclip_runner` agent with no managed AI connection can run in a plugin environment that provides Runner tasks. The run uses the account installed on the execution host. If the host Runner does not match, the native session retries 3 times and then blocks the task with "Plugin-provided Runner execution operation unavailable". The operator cannot tell what to fix. Proposed solution Refuse these runs in Core before any lease, plugin call, secret resolution, or provider work. Use one stable reason, `ai_connection_required`, for run admission, the setup probe, and agent verification. Show a task notice that names the fix. Alternatives considered A refusal in the plugin. It cannot see the connection at `describe`, which runs first. Its error text is also replaced twice before the user sees it. A new structured plugin error code would still not cover `describe`. ## What Changed - Add `server/src/services/plugin-runner-ai-connection.ts`. It holds the `ai_connection_required` code and message, the probe result, and the manifest lookup for a Runner-task driver. The lookup moves here from `agent-runner-selection.ts`, so run admission and runner selection use the same rule. - `heartbeat/execution-configuration.ts`: refuse a new `paperclip_runner` run in a Runner-task environment when it has no managed AI connection. This also applies when `homeMode: "preconfigured"` drops the connection. The run fails as `configuration_incomplete` with reason `ai_connection_required` and an action URL to the agent's runtime settings. The check runs before observability, secrets, the managed runtime, the workspace, and the lease. A resumed native run keeps its admitted contract. - `recovery/stranded-notice.ts`: add an "AI connection required" task notice. - `run-failure-report.ts`: treat this pre-dispatch refusal as an owner action, not an unexpected failure. - `adapters/registry.ts`: the unmanaged probe for a Runner-task target now returns a failed `ai_connection_required` check. It calls no task operation. Managed probes still go through `testManagedEnvironment`. - `agent-harness-verification.ts` and `agent-lifecycle/application/failure.ts`: classify the check as `ai_connection_required` with a fixed message. Verification waits for the configuration to change instead of retrying. - Docs: `PLUGIN_RUNNER_EXECUTION.md` states the requirement. `PLUGIN_ENVIRONMENT_TASKS.md` no longer mentions installed accounts. `agent-config-ui.md` notes the requirement. - Tests: the "plugin-owned account" heartbeat test described the behavior that this change removes. It now checks the assembled heartbeat refusal: no plugin call, no native boundary, no adapter, no workspace files. ## Verification - `npx vitest run` in `server/` for these files. All tests passed: - `src/__tests__/heartbeat-execution-configuration.test.ts`. New cases: refused without a connection, refused for a preconfigured home, managed passes, resumed native input passes, non-task plugin driver passes. The refused cases also check that observability, secrets, and the managed runtime did not start. - `src/__tests__/heartbeat-plugin-environment.test.ts`. One test, "inherits the instance default environment across companies…", also fails on unmodified `master` in my environment. This change does not touch it, because its driver does not declare `supportsTasks`. - `src/adapters/registry.test.ts`, `src/__tests__/agent-harness-verification.test.ts`, `src/modules/agent-lifecycle/__tests__/diagnostics.test.ts`, `src/services/recovery/stranded-notice.test.ts`, and `src/services/__tests__/run-failure-report.test.ts`. - 17 more suites that use plugin Runner-task environments, with 705 tests: `plugin-runner-task.integration`, `plugin-warm-tasks.integration`, `task-creation-recovery.integration`, `native-startup-cleanup-recovery`, `environment-task-authority.integration`, `plugin-task-credential-writeback`, agent-lifecycle `lifecycle`, `setup`, and `environment-plugins`, `workspace-preparation`, `agent-runner-selection`, `agent-harness-verification-provider-api`, `agent-harness-verification-remote`, `plugin-environment-driver-seam`, `environment-task-runtime`, `agent-test-environment-routes`, and `adapter-registry`. - `pnpm typecheck` in `server/` reported no errors in the changed files. Its only errors came from sandbox-provider packages that my local install did not link. - Not run: `plugin-runner-probe.test.ts`, because it needs a Cargo build. This change does not modify `probePluginRunnerEnvironment`. Also not run: the full `pnpm test:run` and `pnpm build`. CI runs them. ## Risks - Behavior change: an agent in a plugin Runner-task environment with no managed AI connection now stops before dispatch. Before, it ran with the host's installed account. The agent's owner must choose an AI connection. This includes agents with `homeMode: "preconfigured"` in these environments. - Runs that were already admitted and resume from saved native input are not refused. They keep the earlier behavior until they finish. - The check reads the plugin manifest on each new run admission for plugin environments. This is the same lookup that runner selection already does. - This pull request and #19326 both edit `stranded-notice.ts` and `failure.ts`. The changes are additive, but whichever merges second needs a small rebase. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (file edits, shell, test runs). Default context window, extended thinking on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 14:59:40 -08:00
Nicky LeachandClaude Opus 5.5	a6ab6a301c	fix(server): retry a run when its remote Runner is out of date (#19326 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - An agent can run on a remote Runner that an environment plugin provides > - A release can update the app before the remote Runner, so for a short time the app sees an older Runner > - In that window the plugin can refuse the task, but Core cannot tell this refusal from other errors > - Today the run fails as `setup_failed` with no retry, and the task goes to board recovery. Before #19246, it was also reported as a project authentication conflict > - Agent verification also counts the refusal as a failed harness test, and it parks the agent after three tries > - This pull request gives the plugin a stable error code for a Runner mismatch, and Core waits and retries when it gets that code > - The benefit is that a release no longer blocks tasks or parks agents, and a real authentication conflict keeps its own clear notice ## Linked Issues or Issue Description Refs #19246 (it stopped reporting every precheck error as a project authentication conflict) Refs #19249 (the same bounded bootstrap retry for a stopped plugin worker) Refs #19302 (shares `environment-task-error.ts`; its diagnostic envelope is unchanged) Refs #19172 (related open PR: it reports an undescribable plugin Runner as a retryable check in the same precheck) Refs #18667 (a real `ai_connection_incompatible` conflict, which now gets its own notice) What happened? A run started while its remote Runner was older than the app. The environment plugin refused the task because the installed Runner did not match the revision it expects. Core recorded a generic setup failure, scheduled no retry, and blocked the task. Before #19246, Core reported this as a project authentication conflict. An agent in `verifying` counted the same refusal as a failed harness test. After three tries it stopped verification until a person chose Retry. Expected behavior Core treats a Runner mismatch as temporary. A run retries without blocking its task. A `verifying` agent checks again until the Runner matches. Only a real `ai_connection_incompatible` result blocks the task, with a notice that names the conflict. Steps to reproduce 1. Use an agent on a plugin environment whose installed remote Runner revision differs from the revision the plugin expects. 2. Wake the agent on a task, or put a new agent through verification. 3. The run fails with `setup_failed` and the task is blocked. Verification parks after three tries. Paperclip version or commit master at `377e98f1a2`. Deployment mode Authenticated deployment with a plugin environment that provides a remote Runner. ## What Changed - Plugin SDK: add `PluginEnvironmentTaskError` and `ENVIRONMENT_TASK_ERROR_CODES` with one code, `runner_revision_mismatch`. The worker RPC host sends it as structured `environmentTask` error data with the schema `paperclip/environment-task-error-code/v1` (`environmentTaskErrorCodeData`, read with `readEnvironmentTaskErrorCode`). This uses the same envelope method as the other environment RPC errors. Core reads only the code, never the message. The source refusal diagnostic from #19302 keeps its own schema and API unchanged, and it takes precedence on a source `submit`. - `executeEnvironmentTask` still replaces each worker error with a generic error. After the #19302 diagnostic check, it now keeps only that closed code. - Run setup: a Runner mismatch before provider dispatch gets the error code `runner_revision_mismatch`. Setup schedules a bounded retry with its own reason, `runner_revision_wait`, and budget (10 attempts, 60 s apart). The reason is in the same resource-wait lane as workspace and AI connection waits, so a wait does not spend the failure retry budget. The guards are the same as for `plugin_worker_unavailable`. Recovery and legacy reconciliation treat the code like the other bootstrap-only codes. `scheduleBoundedRetryForRun` uses the Runner wait reason and budget for a mismatch run when a caller gives no reason, and the recovery sweeper reports exhaustion from that budget (`failedRunRetryBudgetSpent`). An answered interaction that hits a mismatch also waits for the Runner, and its successor keeps the resolved interaction continuation identity. - Move the managed project auth classification from `heartbeat.ts` to `classifyManagedProjectAuthFailure` in `run-preparation.ts`. Only a found `ai_connection_incompatible` conflict becomes a configuration failure. All other errors keep their own code. - Recovery notices: `ai_connection_incompatible` now has its own notice instead of the "secret/env bindings" text. An exhausted Runner wait gets the notice "Remote Runner is out of date". - Agent verification: the Runner probe and the managed project auth precheck report a mismatch as a retryable check. `assertHarnessTestPassed` maps it to the new lifecycle code `runner_revision_mismatch`. That code never counts toward the three-failure park. The agent checks again every 60 s and becomes ready when the Runner matches. - Document the error code in `doc/plugins/PLUGIN_ENVIRONMENT_TASKS.md`. ## Verification - `cd packages/plugins/sdk && npx vitest run tests/environment-task-error-code.test.ts tests/environment-task-error.test.ts tests/worker-rpc-host.test.ts`: the code crosses the real `environmentTask` RPC. A plain error with the same message gets no code. Other methods get no envelope. The #19302 diagnostic tests still pass. - `cd server && npx vitest run src/services/heartbeat/run-completion.test.ts`: a fresh or wrapped mismatch gets `runner_revision_mismatch` and the runner retry budget, with no configuration failure. A started, untrusted (message only) or native run keeps `setup_failed` with no retry. A real `ai_connection_incompatible` stays `configuration_incomplete` with no retry. Other errors, such as an unreachable execution host, are not reported as an authentication conflict. - `npx vitest run src/services/execution-recovery-attempt.test.ts`: Runner waits do not change `failureRetries`, and earlier failures do not shorten the wait. - `npx vitest run src/services/heartbeat/retries.test.ts src/modules/run-dispatch` (embedded Postgres): a recovery caller without options gets the Runner wait budget and its exhaustion. An answered interaction waits for the Runner, keeps its continuation identity and does not spend failure retries. - `npx vitest run src/__tests__/environment-task-runtime.test.ts`: sanitization keeps only the code. No worker text or cause survives. - `npx vitest run src/modules/agent-lifecycle/__tests__/verification-retry.test.ts` (embedded Postgres): a `verifying` agent keeps checking after five mismatches. It never gets `retryRequired`. It becomes ready when the next check passes. - `npx vitest run src/services/native-runtime/plugin-runner-probe-runner-mismatch.test.ts src/__tests__/agent-harness-verification-remote.test.ts src/services/recovery/stranded-notice.test.ts src/services/recovery/service.pause-durability.test.ts src/services/legacy-execution-recovery.test.ts`. - `npx tsc --noEmit` passes for `packages/plugins/sdk` and `server`. - One test in `src/__tests__/heartbeat-process-recovery.test.ts` ("redacts opaque environment-bound credentials from Sentry diagnostics") fails on my machine. It fails the same way on the base commit without this change. - I did not run the full `pnpm test:run`, `pnpm -r typecheck` or `pnpm build`. CI covers them. ## Risks - Low to medium. The change adds one new setup failure code and one lifecycle code. Other failures keep their current behavior. - A Runner mismatch must come from a plugin that throws `PluginEnvironmentTaskError`. Until an environment plugin does that, behavior is the same as today. - The run retry budget is 10 minutes. If a Runner update takes longer, the task goes to board recovery with the Runner notice. - A remote Runner run without a managed AI connection reaches the Runner check only after native dispatch. That configuration is not supported, so this change does not handle it. Such a run keeps the native session retry (3 attempts, 30 s apart). - #19172 edits the same precheck call in `agent-environment-test.ts`. Whichever PR merges second needs a small rebase. Keep the mismatch check before its generic retryable check. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through Claude Code with tool use (code search, shell and test runs). Reasoning mode: standard agentic coding. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 14:34:23 -08:00
DottaandClaude Opus 5.5	da3084a05d	fix(recovery): do not race a just-accepted interaction's own continuation wake (#19332 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents ask humans for decisions through thread interactions, such as a plan approval > - When a human accepts an interaction, the accept route enqueues a continuation wake for the agent > - A recovery sweep also re-wakes the agent when an accepted interaction appears to have no continuation > - The sweep can read the issue in the short gap between the accept and the accept-time wake, so it queues a second continuation for the same approval > - That redundant wake carries the interaction receipt, so the composer queue shows an immutable "Accepted: <plan>" item that the user cannot edit or discard > - This pull request makes the sweep wait a short grace window after resolution before it treats an accepted interaction as stranded > - The benefit is that one approval produces one continuation, and the queue state agrees with the approval ## Linked Issues or Issue Description No public issue exists. This bug was found in a hands-on onboarding test on a staging cloud workspace. What happened? After the user approved the plan, a queued item stayed in the composer queue. The user could not edit or discard it. The same item stayed. Approval did not create a new item each time. Evidence from the staging workspace (wake diagnostics and run records): - The plan approval resolved at 17:42:05.692. - The accept-time wake and its run were created at 17:42:05.992. That run delivered the approval and succeeded at 17:42:51. - The recovery sweep (`source: issue.interaction_continuation_recovery`) requested a second wake for the same approval at 17:42:06.201. Its `retryOfRunId` fell back to the interaction's source run. This shows that the sweep saw no run after the resolution, so it read the issue before the accept-time wake existed. - The redundant wake was deferred behind later runs and a busy shared workspace. It ran only at 18:19. During that time, `GET /issues/:id/queued-comments` returned it as an immutable interaction entry ("Accepted: Approve storefront prototype plan"). Expected behavior One accepted interaction produces one continuation. A delivered approval does not stay in the composer queue. Steps to reproduce 1. An agent asks for plan approval on a task. 2. A user accepts the plan. 3. The stranded-issue recovery sweep runs within the moment between the accept and the accept-time wake enqueue. 4. The sweep queues a second continuation that carries the same approval. The composer queue shows it as an uneditable item until that duplicate run finishes. ## What Changed - `server/src/services/recovery/service.ts`: the accepted-interaction recovery branch now skips the requeue while the interaction resolved less than `INTERACTION_CONTINUATION_RECOVERY_GRACE_MS` (2 minutes) ago. The check applies only when no successful run has delivered the interaction yet. All other recovery branches for the issue keep their behavior. - The skip reports the grace deadline through `onDeferredDeadline`. The sleep-aware recovery scheduler then rechecks the issue at that deadline, so a lost accept-time wake still recovers without another write. - `server/src/__tests__/heartbeat-process-recovery.test.ts`: two regression tests. - `does not race a just-accepted plan approval before its accept-time wake is visible` reproduces the staging race and is the red/green test. It also asserts that the skip registers the deadline, and that a sweep after the deadline requeues the approval. - `does not requeue an accepted plan approval whose accept-time wake is deferred behind another run` guards the deferred accept-time wake case. This test already passed on `master`. - `dispatches the onboarding first task once the user answered the opening card` now seeds an answer 3 minutes old instead of "now". The test checks that a lost continuation is requeued. An answer inside the grace window is not yet lost, so the older seed keeps the test's intent. Related: the Slack connector cancel bug from the same onboarding test (a stuck steering item after the user declined the connector) is likely the same redundant plan-approval wake. At 17:52:10, right after the decline, the item still queued was a retry of this duplicate wake (it carried the plan approval, not the Slack interaction). This PR removes the duplicate, so it most likely covers that symptom too. That bug has its own investigation. ## Verification Red, on `master` code with only the new tests added: ``` $ cd server && npx vitest run src/__tests__/heartbeat-process-recovery.test.ts -t "plan approval" × does not race a just-accepted plan approval before its accept-time wake is visible 1029ms FAIL src/__tests__/heartbeat-process-recovery.test.ts > heartbeat orphaned process recovery > does not race a just-accepted plan approval before its accept-time wake is visible AssertionError: expected 1 to be +0 // Object.is equality Test Files 1 failed (1) Tests 1 failed \| 3 passed \| 411 skipped (415) ``` Green, with the fix (the final commit also checks the registered deadline and the requeue after it): ``` $ cd server && npx vitest run src/__tests__/heartbeat-process-recovery.test.ts -t "plan approval" Test Files 1 passed (1) Tests 4 passed \| 411 skipped (415) ``` Neighbouring interaction-recovery tests on the final commit (`-t "plan approval\|accepted interaction\|answered question\|answered interaction"`): 9 passed and 1 failed. The failing test, `schedules bounded retries for failed accepted interaction continuation wakes`, is a timing flake. It uses a resolution time in March 2026, so the grace window does not apply to it. It failed 3 of 3 runs with the original `service.ts` on the same machine under load, and it passed 1 of 2 runs with the fix. - `pnpm --filter @paperclipai/server... typecheck`: passes. - `-t "onboarding first task\|plan approval"` on the final commit: 8 passed. - CI runs the full `heartbeat-process-recovery` suite in the serialized server shards. ## Risks - Low risk. If an accept-time wake is really lost, recovery now re-wakes the agent up to 2 minutes later than before. The sweep already runs on an interval, so this adds at most one sweep of delay. - The grace window does not change behavior when a successful run already delivered the interaction. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code with tool use (shell, file edits, browser inspection of the staging workspace API). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 15:47:46 -06:00
DottaandClaude Opus 5.5	cb5d7e440c	fix(server): retry setup when the plugin worker was not running (#19249 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Remote agent runs call a sandbox provider plugin, such as Daytona, through a plugin worker process. > - Sometimes that worker is not running. It can be stopped, restarting after a crash, or retired by the host. > - Then the host refuses the call before it sends anything to the worker. > - Setup recorded this refusal as a generic `setup_failed`. It scheduled no retry, so the task became stranded and needed a person. > - This pull request types the refusal and gives it its own transient setup code. The bounded bootstrap retry then runs. > - The benefit is that the task continues on its own when the worker is available again. Users do not see a blocked task. ## Linked Issues or Issue Description Refs #19150 (removed one trigger for the worker retirement) Refs #19245 (restarts a worker after a fail-closed retirement) What happened? A plugin worker stopped. Each run that then called `environmentExecute` during setup failed with `Cannot call "environmentExecute" — worker for "<plugin>" is stopped`. The run got `setup_failed`, no retry was scheduled, and the task moved to board recovery. On one staging instance, 15 runs failed this way in 12 minutes. Expected behavior No provider work started, so the failure is safe to retry. The run must retry within the bootstrap budget, as it does for a reusable sandbox resume failure or a preparation timeout. Steps to reproduce 1. Stop the sandbox provider plugin worker, or let the host retire it. 2. Wake an agent whose run needs that provider during setup. 3. The run fails with `setup_failed`. No retry is scheduled. Paperclip version or commit master at `76e43852`. Deployment mode Authenticated cloud deployment (staging). ## What Changed - Add `PluginWorkerUnavailableError` with the code `plugin_worker_unavailable` (`server/src/services/plugin-worker-unavailable-error.ts`). - The plugin worker manager now rejects with this error when the worker is stopped, crashed, restarting, not running, or not registered. The message text does not change. - Setup failure handling maps this error to `plugin_worker_unavailable`, but only before provider dispatch and only for a legacy run with no native session or runner instance. These are the same guards as for the sibling transient codes. - `isTransientWorkspacePreparationCode` includes the new code, so setup schedules the bounded bootstrap retry. - Legacy execution recovery and the recovery service treat the code like the other bootstrap-only codes. The stranded-task notice has a title and a next action for it. ## Verification - Red: the new `run-completion` test failed on the unchanged code. A fresh refusal got `setup_failed` and no retry. - Green: the same test passes. A fresh refusal gets `plugin_worker_unavailable`, `providerWorkStarted: false` and a bounded retry. A started, untrusted (message-only) or native run keeps `setup_failed` and gets no retry. - New worker manager test: a stopped handle and a missing worker both reject with the typed error. The call is not sent, and `onWorkerSettled` runs once. - `pnpm --filter @paperclipai/server exec tsc --noEmit -p .` passes. - Related suites pass: run-completion, plugin-worker-manager, legacy-execution-recovery, heartbeat-retry-scheduling and recovery (17 files, 439 tests). ## Risks - Low risk. The retry uses the existing bounded bootstrap budget. When the budget is exhausted, the existing board recovery action takes over. - The retry applies only before provider dispatch. A worker that stopped after work started keeps the old behavior. - A wrapper that drops the error `cause` keeps the old `setup_failed` behavior. It does not cause a wrong retry. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 09:05:14 -06:00
DottaandClaude Opus 5.5	e37ab23dbc	fix: hold an agent's new work while its provider usage limit is active (#19246 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Routine-driven agents run on the native Runner through ACP harnesses such as Claude > - When the agent's provider account runs out of allowance, every new wake for that agent fails the same way > - The ACP harness reported the exhausted allowance as a generic "terminal limit failure" with an `unknown` category, so Paperclip could not tell it from a context or turn limit > - Paperclip recorded a reset time only for the failing run's own startup retry; other wakes of the same agent kept dispatching, each spent about two minutes on workspace setup, and then failed > - This pull request classifies the limit, keeps its reset time, and holds the agent's new work until that reset > - The benefit is that a limited agent stops burning runs, shows a clear "waiting for usage limit reset" state with a time, and resumes its queued work after the reset ## Linked Issues or Issue Description Refs #13148 Refs #13379 What happened? A routine-driven Claude agent on the native Runner failed 29 times in about 15 hours. 20 runs failed with `native_provider_terminal_failed` and "ACP agent reported a terminal limit failure." The recorded provider failure had category `unknown`. 6 runs failed with `provider_session_open_rejected`. Their `session.open` result is `command_execution_failed` with classification `provider_rate_limit`, and the provider failure is HTTP 429 `rate_limit` ("would exceed your account's rate limit"). The startup retry of one such run recorded a cooldown until one hour later. A new wake for a different task of the same agent dispatched 16 minutes later, spent two minutes on workspace setup, and failed with the same 429. Separately, 4 runs failed with "Project authentication conflicts with this agent's managed AI connection". That message did not say which file or setting conflicted. The same message was also shown when the check could not finish. What did you expect? A provider usage limit is classified distinctly. Paperclip records the reset time when the provider reports one. New work for that agent waits for the reset with a visible reason. Queued work resumes after the reset, and no doomed run starts. A configuration conflict is reported only when one is found, and the message names the file to fix. Steps to reproduce 1. Use a native Runner agent with a Claude ACP harness whose account is rate limited or out of usage. 2. Let a routine or assignment wake the agent on several tasks. 3. See each run fail with a generic terminal limit failure or a `session.open` rejection, and see new runs keep starting. ## What Changed - Runner: The ACPX turn now receives the typed AIR session failure (`onTerminalSessionFailure`) through the runtime host and the ACPX port. The sidecar adds the provider's title to the provider failure message. Only a rate-limit or quota title becomes `rate_limit` or `quota_exhausted`. Context, turn and configured budget limits stay `unknown`. The failure details stay private. - Server: A native `turn.failed` with a `rate_limit` or `quota_exhausted` provider failure is now `native_provider_usage_limit`, the same code as the Codex usage-limit terminal. - Server: `agentProviderLimitCooldown` (extracted from the startup retry) also reads a reset clock in the provider's wording, such as "resets 2am (UTC)". The clock parser moved from the recovery service to `provider-quota-reset.ts` without changes. - Server: New `heartbeat/provider-limit-wait.ts`. Before any workspace or provider work, a queued run for an agent with an active provider limit is cancelled as `provider_limit_wait`. The run records the reset time, whether the provider reported it, the category and the run that reported the limit. Paperclip schedules one retry of the same wake at the reset through the existing AI connection wait lane. That lane does not consume the failure retry budget. Manual runs, recovering runs, conversation turns and pool-routed connections are not held. - Server: The execution projection shows "Waiting for usage limit reset" with the reset time and the next step. - Server: Only a found project authentication conflict is a configuration failure. Its message names the project file and the settings to remove. A check that cannot finish (timeout or another exit code) is now an ordinary failure, not a configuration conflict. - Docs: `doc/execution-semantics.md` describes the agent-level limit wait. ## Verification Red before the fix, green after: - `server/src/__tests__/heartbeat-provider-limit-wait.test.ts`: An automation wake after a 429 `session.open` rejection dispatched the adapter before the fix. Now the run is cancelled as `provider_limit_wait` and a scheduled retry exists at the reset. After the reset passes, the retry is promoted and the work succeeds. A manual run, a later success, expired evidence and an unclassified terminal still dispatch. - `native-startup-retry.integration.test.ts`: "reads the reset clock from an ACP terminal limit failure's provider wording" (it fell back to one hour before). - `native-session-executor.test.ts`: An ACP `turn.failed` with a typed `quota_exhausted` or `rate_limit` failure is a usage-limit terminal; `unknown` is not. - `packages/paperclip-runner/src/drivers/provider-failure.test.ts`: A typed AIR limit title is kept and classified; context, turn and budget limits are not; details do not leak. - `execution-projection.test.ts`, `ai-connections.test.ts`: The wait label, and conflict versus unfinished check. - Safety cases: `runtime-host.test.ts` and `codex-runtime-adapter.test.ts` forward the callback. The existing startup retry, provider capacity, recovery classification, legacy reconciliation, retry scheduling and execution configuration suites pass. Commands: - `cd server && npx vitest run src/__tests__/heartbeat-provider-limit-wait.test.ts src/__tests__/ai-connections.test.ts src/services/native-runtime/ src/services/recovery/ src/services/execution-projection.test.ts src/services/legacy-execution-recovery.test.ts src/__tests__/heartbeat-retry-scheduling.test.ts src/__tests__/heartbeat-execution-configuration.test.ts` - `cd packages/paperclip-runner && npx vitest run src/drivers/provider-failure.test.ts src/drivers/acpx/runtime-host.test.ts src/drivers/acpx/codex-runtime-adapter.test.ts src/cli/acpx-runtime-sidecar.test.ts` - `tsc --noEmit` for server and paperclip-runner; the ACPX sidecar and provider-failure contract checks pass. I did not run a live provider limit on staging. The live evidence is the staging run records for the affected agent, which I read only. ## Risks - The limit evidence is per agent, not per account. If the board changes the agent's AI connection, automatic wakes can still wait until the old reset (at most one hour when no reset was reported, and at most one day). A manual run is not held, and one successful run clears the evidence. - A new wake that arrives during a limit is now cancelled and retried at the reset, instead of failing. The task stays in its status, and the run shows the reset time. - The Runner change takes effect when the provider pack is rebuilt and deployed. Until then, the server still holds new work using the existing `rate_limit` evidence from `session.open` rejections. - A turn terminal with a typed rate limit or quota is now `native_provider_usage_limit` instead of `native_provider_terminal_failed`. Both are terminal and board-owned. The recovery text now asks the board to restore provider usage capacity. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, browser reads of staging run records) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 08:57:55 -06:00
DottaandClaude Opus 5.5	2ce5f03cd8	fix(workspaces): name remote restore failures and leave a board recovery action (#19207 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A remote task restores its saved workspace into a sandbox before the agent starts. > - When that restore fails, the run records only `remote_workspace_operation_failed: operation=restore exit=1`. > - The run then fails as `setup_failed`. No retry starts, and a `blocked` task gets no recovery action. > - An operator cannot tell which step failed, and nobody owns the next step. > - This pull request makes the sandbox scripts name their failed step in a closed grammar, and it gives a restore failure a bounded retry or a named board recovery action. > - The benefit is a visible cause and a clear owner. Saved work stays protected. ## Linked Issues or Issue Description Refs #19152 and #19187. Those changes name copy-back (save) failures. Refs #18852 and #19195. Those changes added the bounded bootstrap retry that this change reuses. What happened? A board reply woke an agent on a `blocked` parent task. The run failed during setup 13 seconds later. The only run event was `remote_workspace_operation_failed: operation=restore exit=1`. The run had no workspace operation row. The task stayed `blocked`, and its recovery actions showed `active: null`. The same opaque family also appeared for workspace preparation (`exit=1` only). Expected behavior The run names which restore step failed, without command output or paths. A failure before provider work uses a bounded retry when a retry can help. Otherwise, and when the retry budget is spent, the board gets a named recovery action. The saved version is never changed or overwritten. Steps to reproduce 1. Save a remote workspace, then start a new run on a different allocation. 2. Put a folder at the published workspace path in the sandbox before restore. 3. Observe that the base code records only `operation=restore exit=1`, then `setup_failed`, with no recovery action for a `blocked` task. Paperclip version or commit Base: `87dfba6818`. Deployment mode Legacy runtime with sandbox (Daytona) remote workspace storage. ## What Changed - The prepare, repository, and restore sandbox scripts track their current stage. On failure they print `remote_workspace_<op>_failed: stage=<stage>` plus closed fields: `reason=` (the script's own snake_case refusal), `archive=` (the archive helper's fixed reason), `code=` (errno), `disk=full`, `git=` (a fixed label chosen from Git output), and the existing `exit`/`signal`/`program` fields. - The server accepts that text only when it matches a strict grammar for the same operation. Other output is dropped, as before. - Provider refusals during the restore upload and restore command are recorded by closed labels, like copy-back. - A failed saved-workspace restore raises `RemoteWorkspaceRestoreError` with the server step (`snapshot_lookup`, `snapshot_download`, `sandbox_restore`). It is retryable only when the owner re-reads the same writer, epoch, and saved version with no unsaved work, and the sandbox did not name a refusal reason. - Setup failure handling classifies a pre-dispatch legacy restore failure as `remote_workspace_restore_failed`. A retryable failure uses the existing bounded bootstrap retry. A non-retryable or exhausted failure records a board-owned recovery action with the redacted cause, when stranded-work recovery does not scan the task status (for example `blocked`). - Stranded-work recovery gets a named notice for this code. Generic continuation recovery does not grant it a second budget, and legacy reconciliation does not invent provider actions for it. - `doc/execution-semantics.md` documents the contract. ## Verification - Red: the new transport tests failed on the base code with `remote_workspace_operation_failed: operation=restore exit=1` (and the same for prepare and repository). The new completion tests failed because the run became `setup_failed` with no retry and no recovery action. - Green: `npx vitest run src/services/remote-workspace-transport.test.ts src/services/heartbeat/run-completion.test.ts src/services/remote-workspaces.integration.test.ts src/__tests__/heartbeat-retry-scheduling.test.ts src/services/legacy-execution-recovery.test.ts src/services/recovery src/services/remote-workspace-save-concurrency.test.ts src/services/native-runtime/native-workspace-finalizer-failure.test.ts src/__tests__/native-workspace-finalization-work.test.ts` passes (563 tests). - The transport tests use real sandbox commands. They prove an occupied folder reports `stage=inspect_root reason=occupied_root` and keeps its files. They prove a wrong manifest reports `stage=extract archive=manifest_mismatch`. They prove a missing branch reports `stage=checkout_branch`, and a changed origin reports `stage=verify_origin`. Forged output with a path is dropped. - The real-PostgreSQL cold-restore test proves an occupied root gives a non-retryable `sandbox_restore` failure and keeps the unknown files. - The completion tests cover retried, exhausted, refused, post-dispatch, and `in_progress` cases. - `tsc --noEmit` for the server reports no errors in the changed files. ## Risks - A retry keeps the same restore path. If the cause is deterministic but unnamed, the run uses its two retries before the board action. Named refusals skip the retries. - The board action is recorded only for statuses that stranded-work recovery does not scan, and only when no other action is active. This prevents a second recovery identity for one failure. - Script stderr grammar is new. Old sandboxes run the script that the server sends, so no runner update is necessary. - No migration, deployment, or credential change. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use, code execution, and browser tools for read-only staging evidence. Context window and reasoning mode are host-managed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 02:40:01 -06:00
DottaandClaude Opus 5.5	c656b95b6b	fix(environments): retry a refused sandbox resume with the bounded bootstrap budget (#19195 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A warm remote agent keeps its sandbox after a turn, so a follow-up turn resumes the same sandbox and its files > - The sandbox provider auto-stops an idle sandbox (Daytona: 15 minutes by default). The warm native idle window is also 15 minutes > - A follow-up that reaches the sandbox during that stop gets a provider refusal. The host keeps the sandbox ("the lease was preserved and no replacement was created"), which is correct > - But setup then records a generic `setup_failed`. No retry is scheduled, no cause is recorded, and the reply that woke the agent is effectively dropped > - This pull request gives that refusal its own failure code and routes it through the existing bounded bootstrap retry. The retry resumes the same recorded sandbox > - The benefit is that a follow-up continues a few seconds later on the same saved sandbox, instead of stopping the task with no next action ## Linked Issues or Issue Description Refs #18852 (the bounded bootstrap retry that this change reuses). Related: #19152 and #19187 (copy-back causes), #19178 (Daytona errored-sandbox recovery before stop-only cleanup), #19182 and #19180 (recovery holds). This change does not overlap with them: they cover cleanup, copy-back, and recovery after a run. This change covers a resume refusal before provider work starts. What happened? On a staging soak, two follow-up turns failed at setup after a successful first turn: - A board reply woke the agent. The run failed after 4.9 seconds with `setup_failed`: `Reusable sandbox lease "…" could not be resumed; the lease was preserved and no replacement was created.` The issue stayed in review with no recovery action. The reply was not delivered. - An earlier run on another task failed with the same message. Both resume attempts started 15 minutes 8–25 seconds after the previous turn released the sandbox. The first turns used `keep_running` with a warm idle timeout of 900,000 ms. The Daytona default `autoStopInterval` is also 15 minutes. So the resume reached the sandbox while the provider was stopping it. The provider refused within seconds; this was not a long resume timeout. The provider cause was not recorded, so this timing is the strongest evidence. The exact provider state remains an inference. Expected behavior A refused resume before provider dispatch is a temporary infrastructure condition. The host retries it with a small, bounded budget. Each retry resumes the same recorded sandbox. A refusal does not prove that the sandbox is lost, so the host never destroys it or creates a replacement. Only an explicit provider not-found or a workspace identity mismatch with a verified backup permits replacement (unchanged). The run records why the provider refused, without provider text. Steps to reproduce 1. Run a warm remote turn that keeps its sandbox. 2. Make the provider's `environmentResumeLease` throw a non-transient error (for example, a 400/409 while the sandbox stops). 3. Wake the agent with a reply. 4. Before this change: the run fails with `setup_failed`, no retry is scheduled, and the error has no cause. Paperclip version or commit Seen on staging at `5b99bdae`. The touched code is unchanged on current `master` (`f7a4c5678f`). ## What Changed - Move `ReusableSandboxResumeError` to `server/src/services/reusable-sandbox-resume-error.ts` (re-exported from `environment-runtime.ts`). Give it the code `reusable_sandbox_resume_failed`. - Add the provider refusal to the error message by closed labels only: `(cause=<label> status=<http> code=<errno>)`. The labels come from the existing transport classifier, now exported as `sandboxProviderFailureLabel`. Provider text, paths, and IDs are never copied. - Carry the provider status across the plugin worker RPC. A JSON-RPC error has only a message and a numeric RPC code. The worker host now attaches a validated `paperclip/environment-resume-error/v1` envelope (HTTP status and an allowlisted errno only) to `environmentResumeLease` failures. The host re-validates it and adds it to the recorded cause. - In setup-failure settlement, find this error through the orchestrator wrapper. Record `reusable_sandbox_resume_failed` only when provider dispatch did not start and the run has no native session or runner identity (the same guard as `remote_workspace_prepare_timeout`). - Add the code to `isTransientWorkspacePreparationCode`. The existing scheduler then gives two retries (30 s, 60 s) with its issue gate, Stop, pause, and dependency checks. - Exclude the code from generic continuation recovery and from legacy reconciliation (no unknown provider actions; bootstrap evidence proves none started), like the template. - Add a specific stranded-recovery notice: "Saved sandbox could not be resumed". - Document the contract in `doc/execution-semantics.md`. ## Verification - Red (before the fix, same tests): 6 failures. The `fresh` case recorded `setup_failed` and did not schedule a retry. The error had no cause. Legacy reconciliation held the run. Generic recovery granted another budget. - Green: `npx vitest run src/services/heartbeat/run-completion.test.ts src/services/legacy-execution-recovery.test.ts src/services/recovery/ src/__tests__/heartbeat-retry-scheduling.test.ts src/__tests__/environment-runtime.test.ts src/services/remote-workspace-transport.test.ts src/services/heartbeat/retries.test.ts` (in `server/`): 619 tests pass. Also `src/services/recovery/stranded-notice` and `src/services/remote-workspace*`: 411 pass. - Safety regressions: - `run-completion.test.ts`: only a pre-dispatch legacy failure with no native identity gets the code and a retry. `started`, `retained` (native), and an untyped copy of the message stay `setup_failed` with no retry. The recorded error contains the closed cause and not the provider path. - `heartbeat-retry-scheduling.test.ts` (real PostgreSQL): two concurrent schedulers across a restart create exactly one successor per attempt. The budget ends after two attempts. Stop, pause, and a new dependency cancel the scheduled retry. Generic stranded recovery creates no extra run after exhaustion. - `packages/plugins/sdk/tests/environment-resume-error.test.ts`: through the real worker RPC, the `JsonRpcCallError` has no status, and the envelope gives 409 and the errno. Provider text does not cross. Other methods get no envelope. Malformed envelopes are rejected. - `environment-runtime.test.ts`: a throwing resume still preserves the exact lease. The runtime does not call destroy or acquire. The resume throws a `JsonRpcCallError` whose message has no status. The thrown message names `cause=provider_error status=409` from the envelope, and not the provider text. - `node scripts/typecheck-low-memory.mjs` passes. `git diff --check` passes. - Not verified: a live staging run at this revision. The original tasks were not retried. This PR is not deployed. ## Risks - Low. The change affects only runs that fail with this typed error before provider dispatch. Each retry resumes the same lease and inherits the source wake context, so a reply is delivered once and no second executor starts. - If the provider cannot start the sandbox for longer than about 90 seconds, the two retries end and the task gets the specific recovery notice. The sandbox and its saved files are kept. - The error message now has a parenthetical cause before the semicolon. No code parses this message. - An issue in review does not get the immediate stranded notice (unchanged existing behavior). The retry, not the notice, delivers the reply in that case. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, browser reads of staging diagnostics). Extended reasoning enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 00:20:21 -06:00
Svetlana ZolotenkovaandClaude Opus 5.5	bce9551a36	fix(recovery): skip successful-run handoff for routine executions (#9129 ) ## Thinking Path > - Paperclip orchestrates AI agents for zero-human companies, and the issue board is the surface those agents work from > - Recovery is the subsystem that notices when an agent run finished but left no next step, and hands the issue back so the work is not silently dropped > - `decideSuccessfulRunHandoff` implements that check: after a successful run it looks for a valid disposition on the issue and enqueues a corrective handoff wake when it finds none > - Routine execution issues (`originKind = "routine_execution"`) deliberately stay `in_progress` between cron fires — one open issue is coalesced across dispatches through the `issues_open_routine_execution_uq` unique index > - Recovery read that intended steady state as a missing disposition, so every successful routine run enqueued a handoff wake, exhausted its single allowed attempt, and left the issue `blocked` with a `source_scoped_recovery_action` that re-woke the assigned agent > - This pull request exempts routine executions from the successful-run handoff, mirroring the existing issue-monitor maintenance skip (`isIssueMonitorMaintenanceRun`) > - The benefit is that routines stop generating self-inflicted recovery churn while genuine stranded issues keep being caught ## Issue _No public issue exists for this yet; describing it inline per CONTRIBUTING.md ("Link Issues or Describe Them In-PR"), following the bug report template._ What happened? After every successful run of a routine execution issue, recovery enqueued a corrective handoff wake because the issue had no terminal disposition. The single allowed handoff attempt was then exhausted and the issue landed in `blocked` with an active `source_scoped_recovery_action`, which re-woke the assigned agent. Observed in production as roughly 27 runs per hour on a single orphan-check routine, which additionally tripped a false productivity review on the assigned agent. Expected behavior A successful routine run that leaves the routine execution issue `in_progress` is the intended waiting state. Recovery should skip the successful-run handoff for `routine_execution` issues and let the routine lifecycle own their disposition. Steps to reproduce 1. Create a routine that dispatches an execution issue on a short cron interval. 2. Let one dispatched run complete successfully without writing a terminal status on the routine execution issue. 3. Observe the corrective handoff wake, the exhausted attempt, and the issue moving to `blocked` with an active source-scoped recovery action. 4. Repeat on the next cron fire — the cycle restarts and the run rate climbs. Paperclip version or commit `master` at the time this branch was cut; the guard is absent on current `master`. Deployment mode / adapters / database Adapter-independent — the path is server-side in `server/src/services/recovery/successful-run-handoff.ts`. Covered in tests against embedded Postgres. ## What Changed - `decideSuccessfulRunHandoff` returns `{ kind: "skip", reason: "routine execution owns its own disposition" }` when `issue.originKind === "routine_execution"`, alongside the existing issue-monitor maintenance skip. - `originKind` was added to the handoff `IssueRow` projection and to the issue select in `handleSuccessfulRunHandoff` in the heartbeat service, so the guard has the field it needs. - Added a regression test in `successful-run-handoff.test.ts` asserting that a `routine_execution` issue is skipped rather than enqueued. Related PRs (dedup search): [#8622](https://github.com/paperclipai/paperclip/pull/8622) touches the same routine-execution uniqueness index from the migration side; [#6527](https://github.com/paperclipai/paperclip/pull/6527) auto-resolves stale source recovery actions, which is the symptom this PR stops producing. No PR was found that implements this guard. ## Verification - `cd server && pnpm exec vitest run src/services/recovery --config vitest.config.ts` — recovery suites pass. - `cd server && pnpm exec vitest run src/__tests__/heartbeat-process-recovery.test.ts --config vitest.config.ts` — passes; 59 tests across both suites. - Regression check: with the `routine_execution` guard removed, the new test falls through to `enqueue` and fails; with the guard it passes. - `pnpm --filter @paperclipai/server typecheck` — clean. ## Risks - Low risk, behavioural only. No schema change and no migration in this change. - The guard is narrow: it keys on `originKind === "routine_execution"`, so ordinary assigned issues keep their existing successful-run handoff behaviour unchanged. - If a routine execution issue ever did need corrective handoff, it would now be skipped. That trade-off is deliberate — the routine lifecycle already owns these issues, and the coalescing unique index guarantees at most one open issue per routine. ## Model Used - Claude (Anthropic), Claude Opus 4.6 via the Claude Code agent harness, extended thinking enabled, with tool use (file edits, shell, ripgrep) against a local checkout. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [ ] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --- Review request (2026-08-19). Rechecked against `master` `a5f3ac6b4`: `decideSuccessfulRunHandoff` skips on `hasActiveRoutineContinuation` (another live run on the same issue) but never on `originKind === "routine_execution"`, so a routine issue coalesced across fires still draws a missing-disposition handoff. Green and merges cleanly. We cannot use the reviewer field as outside contributors, so flagging the recent maintainers of the touched files instead: @cryppadotta, @devinfoley. Happy to close if this is not wanted — no need to reply otherwise. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-04 17:35:34 -06:00
DottaandPaperclip	d8c1b9d4b5	fix(runtime): reclaim failed remote checkpoint retries (#18877 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native recovery must preserve the original execution and provider state. > - A failed remote run can retain its provider PID after heartbeat releases the task execution lock. > - Normal resumption excludes that PID, while restart recovery rejects the released lock. > - Recovery must reclaim that exact execution only after current admission and durable ownership checks pass. > - Failed retries and admission holds need separate finite budgets. Neither can prove a provider stopped. ## Linked Issues or Issue Description What happened? A failed remote native run remains in retryable failure without a task execution owner. Normal resumption and restart recovery both reject it before inspecting the preserved remote provider. The current remote workspace reference format also misses the due-retry routing path. Expected behavior A due exact-checkpoint retry reclaims its original task execution only when current ownership and safety gates permit it. Newer task input prevents an old claim. Lost or ambiguous provider state still prevents replay or replacement. Ineligible recovery eventually becomes a quiet manual reconciliation obligation. Steps to reproduce 1. Start a remote native run and persist its provider checkpoint and encrypted Runner authority. 2. Fail with exact-checkpoint recovery and a retained remote PID. 3. Release the task execution lock at heartbeat completion. 4. Run ordinary resumption and restart recovery after the retry deadline. 5. Observe that both reject the same eligible run before remote inspection. Paperclip version or commit Reproduced from source at `d59791828482`; integrated with upstream `db9be4c5a6`. Deployment mode Remote workspaces with PostgreSQL. Local regressions use disposable PostgreSQL and encrypted checkpoint persistence. Related public repairs #18846, #18854, #18860 and #18841 cover finished per-turn owners, pre-provider failures, unstarted follow-ups and storage migration. They do not reclaim this failed run's released execution lock. Merged #18879 also adds scheduled-resume lock retention and reclaim. This PR keeps that compatibility and requires encrypted Runner authority, canonical source ordering and a charged failed retry before reclaim. Its scheduled-resume regression now uses a real encrypted authority and retains a snapshot-only refusal control. ## What Changed - Share normal admission checks and one guarded execution-binding restore/event helper between normal and remote resumption. - Keep agent-name normalization in a module with no imports. Preserve heartbeat's existing export. Native admission no longer loads heartbeat orchestration for a string key. - Recheck company, assignment, agent eligibility, Stop, pause, approval, dependencies, execution and checkout ownership before reclaim. - Reject newer task input using PostgreSQL creation-time and ID ordering, preserving timestamp precision. - Require the recorded remote target, encrypted Runner authority, exact checkpoint identity and existing controller fence. - Commit task binding, controller lease, run status and recovery event atomically. Recognize both remote workspace reference formats. - Charge failed reattachments against the existing three-provider-attempt budget, including retained-lock crash windows. Active controller adoption preserves its charged attempt. - Use the existing five-retry/30-minute system-recovery policy for admission holds. Ended tasks and newer sources exhaust it immediately. Preserve PID, checkpoint and task bindings, suppress resource settlement, and assign manual reconciliation with executor stop/state explicitly unverified. - Register recovery intent before deadline writes; preserve live/future deadlines quietly and roll back zero-row writes. Record changed hold reasons once and terminal dispositions separately. - Keep historical provider-budget exhaustion behind current admission/source/controller/authority checks. Make an existing orphan-cleanup fixture explicitly due. - Include exact plugin task leases that have no workspace reference. Reuse every reclaim and retry-budget guard, with a bounded company/task/run lease projection. - Share strict manual ownership proof across stale-lock cleanup, adoption, queue release and wake admission. Require termination receipts for every exact remote lease before normal cleanup can release either binding. - Preserve governing Stop through the existing audited cancellation path. Its acknowledgement does not release an unverified provider. Ordinary Retry can wait for exact Stop and file preservation, then queue one continuation. ## Verification - Current head is `5d7baf74aa89d00d7ac6e1493d38a6ce2ae7b7b5`. Its [current CI matrix](https://github.com/paperclipai/paperclip/actions/runs/41782198858) is terminal and clean: 54 successful, three skipped and two neutral contexts. Required `ci / verify` and `ci / e2e` pass. Greptile is 5/5 on this exact head with no open findings. No fresh Paperclip 5/5 assessment is claimed. - The [prior complete CI matrix](https://github.com/paperclipai/paperclip/actions/runs/41754365388) passed on `3e641466d3edcaa6dd5021d979627082d388eb05`, with Greptile and Paperclip both 5/5 and no open threads. - All 233 tests across the six affected heartbeat, restart-isolation, remote-admission, scheduled-resume and session-binding families pass on production-identical `a5104fa89f9cb493091e000bf21ed3858871ae46` under Node 24.22.2 and pnpm 9.15.4. They cover both workspace formats and plugin task leases, competing controllers, exact rollback, Stop, assignment/company boundaries, attempts, work-intent ordering, finite backoff and quiet held state. - The merge preserves #18931's per-run audit-failure isolation and this PR's unchanged-transition hold. Two initial composition cases failed because their setup or expected result described the older contract. The rollback test now expects the exact per-run deferral while retaining full coordinator equality and no work intent. The routine fixture writes a real encrypted checkpoint through the normal owner before reaching the actual routine unique constraint. No production admission guard or rollback assertion was weakened. A separate agent reviewed this exact merge and fixture delta without finding a blocker. - Full recursive `pnpm -r typecheck` and `pnpm build` pass on production merge `89c616fe45ceff75db450f67fd34224b4df7441d`. Subsequent commits change only three test files. Both the two-file `a5104fa` and final `5d7baf` server test-layer typechecks pass. - Four production-identical `a5104fa` probes pass with the rebuilt actual Rust Runner, a scripted ACPX subprocess and encrypted PostgreSQL checkpoints under Node 24.22.2. Workspace-reference and plugin-lease admission each cover surviving and lost providers. Replacement preserves one provider launch, one original side effect, the charged attempt and restored task execution binding. Lost providers reject snapshot recovery without replacement. This is protocol-boundary evidence, not a live Claude or hosted provisioning claim. - Initial `a5104fa` CI found one additional routine integration fixture missing the persisted `exact_checkpoint_resume` mode. Its encrypted checkpoint was already present. The exact case fails locally before repair and passes after recording that mode and expecting the charged provider attempt. Every exact routine conflict, rollback, checkpoint, task binding, successor-count and sibling-startup assertion remains. Final head changes only that fixture. The correct native-server group passes all nine files: 175 executed cases and 14 container-only skips, matching the now-successful native-server CI gate. - The earlier canonical HTTP Retry acceptance passed on the prior reviewed head with real PostgreSQL, the recovery worker and real files. Pending Stop protects both bindings and starts no successor. Exact termination and copyback queue one preserved-workspace continuation. No browser or live-model outcome is inferred from that test. - The predecessor's full local `pnpm test:run` failed in an unchanged Discord Gateway fixture with a 200 ms lease and a 422 initial ownership error. Focused predecessor and untouched-base controls pass; that full-run cause remains unproved. The required command was also started on production-identical `a5104fa`. All six local chat shards passed, covering 1,850 cases. The duplicate local invocation was interrupted with SIGINT while the general-server group was still running after this head's complete CI and the combined repair qualification had passed. The wrapper exited 1 on cancellation, no process retained its test log, and no full local pass is claimed. An earlier attempt to select the native lane through the CI wrapper instead selected local package sharding: 2,631 cases passed and 13 failed. Those separate failures are retained without causal attribution or unrelated source changes; that run is not counted as native-server qualification. The direct native group and current complete CI matrix pass. - Live Claude Opus 5.5 continuation, hosted plugin provisioning, deployment and recovery of historical tasks remain unverified. No original task was retried or mutated for this verification. ## Risks - Restoring a task binding grants remote inspection; it does not prove that the provider survived. - Unknown effects, lost providers and missing semantic results retain their existing reconciliation requirements. - Exhausted admission holds preserve execution bindings and resource evidence for manual reconciliation. They do not authorize a fresh executor or provider/resource release. - The source guard is conservative: newer task input prevents recovery of an older failed source. - No database migration or provider profile change is required. ## Model Used OpenAI Codex, based on GPT-6, with reasoning, tool use, code editing and shell execution. The runtime does not expose the exact model snapshot or context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-04 07:18:07 -06:00
Devin FoleyandPaperclip	4e29f00139	fix(native): keep settled ACPX sessions across Runner upgrades and run scheduled resumes (#18879 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native Runner keeps one ACPX provider conversation across the per-turn runs of a task. Each run cold-attaches the session that the previous run settled. > - The saved ACPX state binds the digest of the launch profile that wrote it. Every deploy ships a new provider pack, so that digest changes. > - After a deploy, the next run of such a task fails `run.attach` with `ACPX durable launch profile digest does not match runner startup`. Every same-run retry fails the same way. > - On one instance this happened seven times. Six tasks used their retry budget and ended blocked, or in progress with no runner. In the seventh, the scheduled same-run resume never ran at all: the task stayed `in_progress` with nothing running, and its monitor did not fire. > - This pull request re-binds a settled session to the upgraded launch profile, types the remaining mismatches so the server stops retrying them, and makes a scheduled remote resume survive the stale-lock sweep. > - The benefit is that a task continues its conversation after a deploy. When it cannot, it stops with a clear next action instead of stalling. ## Linked Issues or Issue Description Refs #18846, #18847, #18854 and #18860 (nearby native recovery changes). Refs #15795, an open change to the same stale-lock sweep function. It changes run teardown, not lock cleanup, so a second merge needs at most a small rebase. What happened? A per-turn native task used provider `acpx` with Claude, driver `acpx_runtime`, on a remote sandbox. 1. Run A succeeded and settled its provider session. The next run with the same build cold-attached it. 2. The instance deployed a new build. 3. A board approval woke run B in a fresh sandbox. Run B failed after about five minutes: `native_session_recovery_failed: Error: PRP command run.attach failed: {... "result":{"code":"command_execution_failed","message":"ACPX durable launch profile digest does not match runner startup"}}`. 4. The coordinator recorded `retryable_failure`, attempt 1, `native_session_interrupted`, recovery mode `exact_checkpoint_resume`, recovery state `resuming_session`, and a retry 30 seconds later. 5. Two seconds after the failure, the stale-lock sweep cleared the task's execution and checkout locks, because run B was `failed`. 6. Forty minutes later the attempt was still 1. The scheduled monitor had not fired. The task stayed `in_progress` with nothing running. Expected behavior A session that the previous run settled continues after a deploy. If it cannot continue, the run stops at once with a clear next action. A scheduled resume either runs or fails visibly. Steps to reproduce 1. Save a suspended ACPX session under one launch profile. 2. Change the sidecar bytes and the provider-pack authority at the same paths. 3. Cold-attach the next run in a new runner. Restore fails with the digest mismatch. 4. For the stall: leave a remote native run `failed` with a due `retryable_failure` coordinator, run the stale-lock sweep, then run restart recovery for that run. It returns `blocked: issue_execution_lock_changed` and writes nothing. The new Rust and PostgreSQL tests run these steps. Paperclip version or commit `master` at `b0c4e345267`. Deployment mode Native Runner, per-turn lifecycle, remote sandbox. The re-bind applies to every ACPX agent with an ordinary startup. ## What Changed ### Root causes - Upgrade failure. `AcpxLaunchProfile::canonical_digest` hashes the provider-pack authority, the sidecar command and arguments, each artifact path and SHA-256, and any candidate admission. For a remote sandbox the authority is the provider-pack manifest digest, which covers the Runner source revision and dist digest. So every deploy changes it. `AcpxDurableState` stores only the digest, and `validate()` rejects any difference. Only a Kimi candidate run binding had a carve-out. - Silent stall. The failure write keeps the task locks for `retryable_failure` and clears them only for `terminal_failure`. The stale-lock sweep treats any `failed` run as cleanable, so it cleared them. A remote retry goes through `claimNativeRestartRecoveries`. That claim requires `issues.execution_run_id` to name the run, so it returned `blocked: issue_execution_lock_changed` with no coordinator write and no deadline. The live recovery queue then dropped the entry. The local claim (`claimNativeSessionResumptions`) already reclaims a free lock, so only remote runs stalled. - Monitor. `noActiveNativeMonitorRun` holds a task's monitor while any native run for it has a coordinator that is not `committed` or `terminal_failure`. Run B's `retryable_failure` held it. That is by design; it ends when the resume runs. ### Runner (Rust) - `rebinds_settled_session` in `acpx_provider_backend.rs`. Restore re-binds a saved session to the current launch profile only when all of these are true: - The lifecycle is `suspended` and an identity exists. There is no active turn, pending runtime request, undelivered provider event, or unconfirmed provider exit. - The startup has no candidate admission. - The saved descriptor's sidecar command and arguments equal this startup's. - Any other digest mismatch keeps failing with the same message, plus `classification: "provider_state_incompatible"` in the durable command result. `DurableRunnerError` carries an optional closed classification for this. - A saved descriptor that names an agent release this build does not qualify (ACPX version, agent server package or version, or command digest) also gets the class. `validate_session()` and its message are unchanged; restore only adds the class. ### Runner (TypeScript) - `runnerRecoveryFailureClass` reads the classification from the authenticated `runnerCommandResult`. The Codex app-server driver (which also drives ACPX through runnerd) and `HarnessDriverBackend` pass it on as `failureClass`. - If replacement after a failed resume is not allowed, `executeNativeSession` throws `NativeProviderStateIncompatibleError` (message unchanged: `native_session_recovery_failed: ...`). If replacement is allowed, the existing path opens a fresh session in the same run, as before. ### Server - `nativeSessionFailureSourceCode` maps that error class to `native_provider_state_incompatible` with `instanceof`, not text. The disposition is permanent: `terminal_failure` at once, the task blocks, and the board owns the recovery action. Its next action says that a Retry continues in a fresh provider session with the same workspace. - `findNativeSessionResumeRun` also returns the barrier's `errorCode`. `rebindNativeSessionCheckpoint` does not resume a source with `native_provider_state_incompatible`. The existing fallback in `buildNativeExecutionWithCheckpoint` then starts a fresh normalized session with the full task bootstrap. - `sweepStaleIssueLocks` keeps both locks while a referenced run's coordinator owns a result-less `retryable_failure`, unless the task is done or cancelled. Its update also requires that no referenced run has a live status at the write, because a resume claim can return the run to running after the status read. - `claimNativeRestartRecoveries` reclaims a free execution lock for a `retryable_failure` resume, like the local claim does, when the checkout lock is free or held by this run and no other run for the task is queued, running or scheduled. It records `native.recovery.execution_reclaimed`. A lock held by another run still blocks. - Docs: `packages/paperclip-runner/docs/durable-recovery.md` and `doc/execution-semantics.md`. ### Safety argument for the re-bind What the binding protects, and what this change keeps: - Candidate admission carries company, model, platform and expiry authority. A candidate startup never re-binds. The Kimi run-binding carve-out and all candidate rejections are unchanged. - The agent that loads the conversation stays exact. `validate()` runs on the saved descriptor before the digest check and requires this build's qualified agent profile: ACPX version, agent server package and version, and `commandDigest`. The saved identity must still match that profile and the provider mode. - In-flight provider work is never re-bound. A turn, runtime request, undelivered event or unconfirmed exit keeps the strict binding, so no old provider process or request continues under a new build. The next launch only loads a settled conversation. - Launch shape. A changed sidecar command path or arguments still rejects. Only the content at the same paths may change. - Artifacts are not trusted from the saved state. The current profile comes from the trusted controller at startup, and session start verifies every artifact's bytes and the restored provider identity, as before. What changes: a new provider-pack authority, or new sidecar or Node bytes, at the same paths no longer invalidates a settled conversation. The saved digest cannot show which part changed, so the rule does not depend on it. I did not add state fields to tell authority changes from artifact changes, because the rule would treat both the same. One existing assertion changes. `released_kimi_cold_restore_rejects_candidate_injection_and_changed_full_profile` expected a settled ordinary Kimi session to reject a changed authority, artifact digest, or saved digest. Those three cases moved to a new positive test. Command, arguments, every candidate grant field, a missing profile and run binding still reject. ### How a task in the reported state recovers after this deploys The task has old-digest saved state, a `retryable_failure` coordinator at attempt 1, and cleared locks. 1. Startup recovery, and the live recovery queue, run restart recovery for the due run. It reclaims the free execution lock and claims the run. The resume runs now. 2. Run B's failed `run.attach` is a durable outcome of B's exact run authority. If the resume replays that outcome instead of executing it again, it fails with the old, untyped text. The remaining attempts then fail the same way, and the run ends `terminal_failure` with `native_session_retry_exhausted`. The task blocks with a board recovery action. This takes about a minute. 3. If the task blocks, press Retry once. Retry selects run B as the source (it has process identity and controller evidence, so the #18860 skip does not apply) and rebinds B's checkpoint, which holds A's provider session. The new run cold-attaches under the new Runner, and restore re-binds the old digest because the session is suspended and quiescent. The conversation continues. Run B's failed attach never wrote the state file, so the state is still A's settled state. 4. Tasks that already exhausted their retries (blocked, or in progress after `native_session_retry_exhausted`) need the same single Retry. Their earlier attempts ran under the old Runner. Wait for the task to show blocked before pressing Retry. A Retry pressed while the resume is running joins that run. ## Verification - Rust, failing first: `settled_session_rebinds_to_the_upgraded_runner_launch_profile` failed on `master` with the production message. `unsettled_or_relocated_session_keeps_the_strict_launch_binding` failed on the missing classification. Both pass now. - `cargo test -p paperclip-runner-core --no-fail-fast`: all 41 test binaries pass. The library has 471 passed and 1 ignored. - `cargo fmt --check`, and `pnpm --filter @paperclipai/paperclip-runner typecheck` (TypeScript, generated contracts, `cargo check`): pass. - Server, failing first: `native-scheduled-resume-lock.test.ts` (PostgreSQL). On `master` the sweep cleared the locks and restart recovery returned `issue_execution_lock_changed`. Both negative controls (another run's lock, a queued run) already passed. All four tests pass now. - `pnpm --filter @paperclipai/server typecheck`: pass. - Runner vitest (`src/native-session-runtime.test.ts`, `src/backends`, `src/drivers/codex`, `src/drivers/provider-failure.test.ts`, `src/live/runnerd-codex-transport*.test.ts`, `src/control-plane/acpx-controller-recovery.test.ts`): 31 files, 1,011 tests pass. New: the typed error when same-session recovery is required, and the recovery class reader and pass-through. - Server vitest: the native executor, status corpus and provider-pack admission suites (1,268 tests) pass. New: the classifier makes the typed error permanent, and the same text stays retryable. Checkpoint selection returns `errorCode`, and an incompatible source starts a fresh session with the same workspace (unit and PostgreSQL). - Server vitest over the recovery area: stale-lock sweep, native resumption, process recovery, failed-workspace recovery, restart recovery (integration, diskless, preparation), controller recovery, startup retry, startup cleanup, recovered-owner follow-up, session resume, runtime selection and recovery deadlines. 926 passed. Four `heartbeat-process-recovery` tests failed in that loaded run. Three pass when I run them alone. `redacts opaque environment-bound credentials from Sentry diagnostics: result` fails the same way on unmodified `master` on this host (a local environment value is redacted). - Review round 1: a new sweep test returns the run to running before the clear. It cleared the locks without the write recheck and keeps them with it. The strict Rust restore test covers an older agent server release and an older ACPX release. The ACPX backend and durable runner unit tests, the lock, sweep, resume projection and controller recovery suites, `cargo fmt --check` and the server typecheck pass. In the second full cargo run, the two `stdio_mcp` relay tests timed out waiting 5 s for a child exit on this loaded host. They do not use the changed code and passed in the first full run. - After the rebase onto the current `master`: the new lock test, failed-workspace recovery, stale-lock sweep and session resume suites pass (189 tests). Server typecheck, `pnpm check:module-boundaries` and `git diff --check` pass. - CI on `929ef4508d2`: in the first run, `isolated-repository-rehome` hit its 15 s test timeout, `idle-completed-history` could not reach its fixture runtime service, and e2e shard 3 was cancelled by a runner shutdown. A rerun of the failed jobs passed. All 55 checks pass. - Not verified: a live remote sandbox deploy, and whether run B's same-run resume replays its recorded `run.attach` failure or executes it again. The Retry path is tested at source selection and checkpoint rebind (server) and at cold attach (Runner). It is not tested end to end through a sandbox. ## Risks - A settled session now continues under new sidecar bytes. If a sidecar change made an old ACPX record unreadable, session start fails as any failed resume does. It does not corrupt state, and the run follows its normal recovery policy. - The sweep now keeps locks for a failed native run with a scheduled resume. If that resume can never be claimed (for example the agent stays paused), the task keeps the lock until the coordinator settles or the task closes. Before, the sweep cleared it, and the resume still never ran. - If another run holds the task's lock, a scheduled remote resume still stays blocked and the coordinator stays `retryable_failure`. This change does not settle that case. It needs a separate decision about superseding the resume. - `native_provider_state_incompatible` is a new failure code. No UI or migration change is needed. Older servers never see it. - No schema migration. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) through Claude Code, with tool use (shell, file edits, test runs). The runtime does not expose the context window or reasoning configuration. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-03 16:30:58 -08:00
DottaandPaperclip	0a1e4c3d0f	fix(workspaces): retry pristine remote setup timeouts (#18852 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A remote task needs a workspace before its model can start. > - Fresh workspace preparation can reach its transport deadline. > - Failure handling records that timeout as a generic setup failure and stops the task. > - An unpublished workspace has no saved task work to repeat. > - This change routes that narrow failure through the existing bounded retry scheduler. > - Saved work, retained executions, and normal admission gates keep their existing protection. ## Linked Issues or Issue Description Refs #18827, which preserves remote timeout diagnostics. This change builds on that merged repair. Related #16727 concerns controller-managed clones and optional partial clones; this change keeps remote clone contents and deadlines unchanged. What happened? A fresh remote workspace reached its 300-second command limit before model execution. The task then stopped with `setup_failed` and no scheduled continuation. A separate full clone completed in 137 seconds on a disposable Daytona sandbox. The original delay remains unexplained. This change repairs the missing recovery path; it does not claim to remove the underlying delay. Expected behavior A fresh, unpublished workspace preparation timeout can use the existing two automatic retries. Retry scheduling remains durable and company scoped. Stop, pause, dependencies, budgets, task ownership, saved work, and retained native execution must still prevent unsafe work. Steps to reproduce 1. Start preparation for a fresh remote workspace with no saved version or published allocation. 2. Make its initial directory/clone command return a transport timeout. 3. Observe that the base code records a generic setup failure without bounded recovery. 4. With this change, the owner validates current workspace authority and preserves a specific bootstrap failure for the normal retry scheduler. Paperclip version or commit Base: `e0276e0166`, including #18827. Deployment mode Native Paperclip Runner with sandbox workspace storage. Verification uses real PostgreSQL, filesystem commands, and a separate live Daytona transport probe. ## What Changed - Recognize only the initial preparation timeout while the same company, writer, and epoch still own a pristine workspace. - Keep restore, additional repository, provisioning, save, changed ownership, dirty workspace, and retained native execution failures outside this recovery policy. - Persist the typed failure before using the existing retry scheduler, admission gates, and attempt budget. - After successful terminal lease cleanup, reclaim an unpublished bootstrap-only reservation even when a provider returns no stop receipt. Preserve receipt requirements for saved/published work, unknown execution, and native identities; do not synthesize termination evidence. - Prevent generic recovery from granting another budget after exhaustion. Keep a specific recovery notice. - Add regressions for persistence, concurrent scheduling, restart, exhaustion, Stop, pause, dependencies, fresh allocation, lost replies, and saved output after cold restoration. - Document the recovery contract. ## Verification - Red: the real-PostgreSQL workspace regression first failed because the timeout had no recovery classification. The completion regression then failed because it became `setup_failed`. A retained-native negative failed before the admission guard. The review follow-up reproduced a failed successor reservation after production lease release returned void. - Green: all 127 transport/archive/real-PostgreSQL workspace cases pass after the review repair. All 12 receipt/void handoff cases pass against the final query. The 141 complete completion/retry/recovery cases and the fresh-allocation selection case pass. Safety cases cover concurrent scheduling across restart, the two-attempt cap, Stop/pause/dependency gates, native identities, saved/published files, active leases, failed cleanup, and cold-restored output. - `node scripts/typecheck-low-memory.mjs` and `pnpm build` pass at `ba96cff771707415e4a5e893c8b3a97499ead25b`. `git diff --check` and the module-boundary check pass. The full local test wrapper completed all 1,835 chat cases across six partitions, then was interrupted during its serial general-server phase after the complete current-head CI suite passed. Its interrupted run is not a full local-suite pass; its log is retained. The owned process tree and database children exited. - The [complete GitHub Actions run](https://github.com/paperclipai/paperclip/actions/runs/41718108375) is successful at this exact head, including all server/workspace/chat partitions, all nine serialized groups, all eight browser shards, build, typecheck, native Runner, package, Rust, and release-policy checks. The first Rust worker shut down during cache restore before tests; one retry passed without source changes. Expected Storybook skips remain skips. - Earlier failures remain accounted for: the first typecheck found two incomplete process-result fixtures, now corrected. An intermediate completion run exposed a test double that returned no promise and a teardown timeout; the complete 141-case suites subsequently passed. The first-head CI had four worker shutdowns and one worker communication loss. A three-second monitor settlement wait failed in CI and locally, then all 21 cases passed unchanged at base and head. An earlier CI retry found a real-process history adoption assertion failure; that exact case passed locally at base and head. Both server groups pass in final-head CI. These timing causes remain unknown. [Earlier CI evidence](https://github.com/paperclipai/paperclip/actions/runs/41713712522). - Live supporting evidence: unchanged production preparation code completed a full repository clone in 137 seconds using Daytona SDK 0.203.0, Node 24.22.2, Git 2.39.5, and Linux x86_64, with 1 CPU, 1 GiB memory, and 1 GiB disk. The approved diagnostic account could not access the original fleet snapshot, so the probe used public `node:24-bookworm`. Both probe allocations were deleted and absence verified. Full task acceptance at the fixed revision remains unverified: the original fleet snapshot and an explicitly configured disposable Codex subscription fixture were unavailable. No CLI login or hosted subscription was copied. - Greptile is 5/5 at the final head with no actionable findings. The Paperclip review finding is repaired, replied to, and resolved. The separate Paperclip Review check remains queued. Its task failed workspace export after publishing the first assessment; the staging service was then observed showing a workspace recovery page after an update failure. Subsequent browser rechecks timed out, including a fresh tab, so the normal recovery control could not be used. Reconcile that existing export after service access returns, then let the saved current-head review request run. That external review is the remaining check; this PR is not yet fully green. - The original hosted task was last verified blocked. This PR has not been deployed and the task has not been retried by this work. Reviewers can run `pnpm test:run`, `node scripts/typecheck-low-memory.mjs`, and `pnpm build`. The remote-workspace integration suite additionally needs an owned disposable PostgreSQL database named `remote_workspace_*test` through `PAPERCLIP_REMOTE_WORKSPACE_TEST_DATABASE_URL`; otherwise that suite is skipped. ## Risks Each retry keeps the existing preparation deadline, so persistent slow clones can consume the bounded budget before intervention. A retry receives fresh allocation admission; it does not replay the command against the old allocation. The workspace owner checks persisted authority before classifying the failure, and normal admission still rechecks that authority. Historical generic setup failures do not gain automatic replay. No database migration, provider-pack update, model fallback, credential change, or deployment is included. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools, code execution, and browser tools. The host does not expose a more specific serving model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-03 15:36:07 -06:00
Devin FoleyandPaperclip	909d86b59a	fix(native): retry runs that fail before provider work (#18854 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runs start the provider through Runner. `session.open` starts the provider session before any prompt. > - When `session.open` fails, the server records the generic `native_session_interrupted`. Recovery evidence then reports `ambiguous_state`, so the first failure blocks the task with no automatic retry. > - The failure is often temporary. On one instance an agent's provider account hit its usage limit. For about 14 hours every later run failed at `session.open`, and 47 tasks stayed blocked. Outside limit windows, the same error also occurs once and a later retry succeeds. > - The server already has a bounded same-run bootstrap retry for failures that are proven to come before provider work. It did not recognize this failure, and it did not consider a known provider limit. > - This pull request classifies a definite `session.open` failure as a pre-session failure. Bootstrap retries back off and wait out a known provider limit. The task is blocked only after the shared budget is used, with a clear reason. > - The benefit is that a temporary startup failure or a provider limit no longer strands tasks on the first failure, and every existing replay guard stays in force. ## Linked Issues or Issue Description Refs #18847. That pull request keeps the provider cause on `session.open` rejections and names typed categories (for example `classification=provider_rate_limit`). This change does not depend on it. With #18847, a typed limit at `session.open` uses the same cooldown path, because the server reads the cause with `providerFailureFromError`. Refs #18622, which added the `ProviderFailure` cause. Refs #13993, an open change about legacy adapter startup backoff; it does not touch the native recovery path. What happened? A native run (`paperclip_runner`) failed at `session.open` before any prompt: `ACPX sidecar command session.open was rejected (retryable=false, classification=session_ensure_failed)`. The server recorded `native_session_interrupted`. Recovery evidence had no checkpoint and no provider event, but the cause was not in the pre-session list, so the mode was `ambiguous_state`. The coordinator went to `terminal_failure`, a board-owned `active_run_watchdog` action was created, and the task was set to `blocked` on the first failure. During a provider usage limit, every later run of the agent failed the same way, so each task was blocked. Expected behavior A run that provably failed before provider work retries automatically, with backoff and a bounded budget. If the agent has a known provider limit, the retry waits for the cooldown. The task is blocked only when the budget is used, and the recovery action explains why. A run where provider work may have started is never replayed. Steps to reproduce 1. Make Runner reject `session.open` for a native run (for example a provider at its usage limit). 2. Observe the coordinator: `terminal_failure`, `recoveryMode: "ambiguous_state"`, `providerSessionEstablished: false`. The task is `blocked`. 3. The new `native-startup-retry.integration.test.ts` reproduces this with a real database and the executor. Paperclip version or commit `master` at `5877d751c0e`. Deployment mode Native Runner with an ACPX Claude provider. The fix is in the server and is not specific to one deployment mode. ## What Changed - `nativeSessionFailureSourceCode` returns a new cause, `provider_session_open_rejected`, when Runner reports that `session.open` failed or was rejected. Those outcomes are definite, and the command runs before any prompt. An `indeterminate` outcome stays `native_session_interrupted`. Earlier, more specific causes (for example a missing runtime package) keep their classification. - `nativeProviderRecoveryEvidence` treats the new cause as pre-session. The durable evidence still decides. A bootstrap retry happens only with no provider session, no provider event and no checkpoint. Otherwise the mode stays `ambiguous_state` and the task is blocked as before. - New module `native-runtime/native-startup-retry.ts` plans each bootstrap retry. It is called only for `bootstrap_retry` mode. - Backoff: the existing `infrastructureRetryDelayMs` (about 30 s, then about 60 s, plus or minus 20 %). The jitter derives from the run and attempt, and it is persisted with `nextAttemptAt`. - Limit cooldown: evidence is a `rate_limit` or `quota_exhausted` provider failure on the current failure (the #18847 shape), or on an agent run that started in the last 24 hours. A `native_provider_usage_limit` or `provider_quota` error code also counts. The cooldown is the provider's `retryAt` (or `retryNotBefore`), or one hour after the failure when no reset time is known. A later successful run of the agent clears older evidence. The cooldown is capped at 24 hours. The retry runs after the cooldown plus the backoff delay. - A failed evidence read is logged and falls back to plain backoff, so it cannot stop the failure from settling. - The budget does not change: three provider attempts, shared with checkpoint resumes and replacements. When it is used, the coordinator records `native_session_retry_exhausted`. The task is blocked, and the board-owned action says: "Provider startup failed on 3 attempts before any provider work began. Automatic retries are stopped." When a limit is known, the message includes the expected reset time. - While a retry waits, the coordinator detail, the recovery action evidence and the scheduled wake record `providerCooldownAt`. The next action names the reset time, and the task projection shows "Retry scheduled". - `PROVIDER_LIMIT_DEFAULT_COOLDOWN_MS` (one hour) moves to `system-recovery-state.ts`. The legacy provider-quota recovery default now refers to it. The value does not change. - A chat retry of an exhausted `provider_session_open_rejected` run is accepted like the other pre-provider causes. The existing proofs still apply. - `doc/execution-semantics.md` describes the backoff, the cooldown and the exhausted state. ## Verification - Red: before the fix, the first new integration case failed. The coordinator was `terminal_failure` with `recoveryMode: "ambiguous_state"`, `originalFailureCode: "native_session_interrupted"` and `providerSessionEstablished: false`, and the task was `blocked`. Four more cases failed the same way. The two never-retry cases passed before and after the fix. - Green: `npx vitest run src/services/native-runtime/native-startup-retry.integration.test.ts` in `server` (embedded PostgreSQL, the real executor, and the real resume claim). 8 cases pass: - a rejected `session.open` is retried with backoff (attempt 1 in 24 to 36 s, attempt 2 in 48 to 72 s), and the task stays `in_progress` with an agent-owned action and a `resume_native_run` wake; - after three attempts the task is blocked, the action is board-owned with the clear reason, and the resume claim refuses another attempt; - a recent limit failure holds the retry until one hour after that failure, and the claim refuses it before the cooldown; - a provider reset time from an earlier run, or from the startup failure itself, sets the cooldown; - a distant reset time holds the retry for at most one day; - a later success, another agent's limit, and an expired limit do not delay the retry; - a durable provider event, or an indeterminate `session.open` outcome, still blocks the task with `ambiguous_state` and no retry. - `native-session-executor.test.ts`: all 1,087 tests pass, including a new classification case for `failed`, `rejected`, `indeterminate` and a different command. - `chat-channels.integration.test.ts -t "requires terminal-safe native ownership for an exact chat retry"`: 23 pass, including the new `session_open` case. That case fails without the list change. - Related suites pass: native session resumption, native provider failure, capacity and overload integration tests, provider failure classification, native status arbiter corpus, native restart recovery. - `npx tsc --noEmit -p server` passes. `git diff --check` passes. - [GitHub CI](https://github.com/paperclipai/paperclip/actions/runs/41713988964) is green on head `32561aa335d`. This includes `ci / verify`, `ci / e2e`, typecheck, build, all server, chat and workspace groups, all serialized server suites, and all Runner checks, which run the native-runtime server tests. In the first attempt, eleven jobs stopped because their runners received a shutdown signal; no test reported a failure. A rerun of those jobs passed. - A local run of the whole `server/src/services/native-runtime` directory was stopped before completion because it runs serially for a long time. CI ran those files. - Runtime dependency: the new work is one indexed query (`company_id`, `agent_id`, `started_at`) with `LIMIT 20`. It runs only when a bootstrap retry is planned or exhausted, not on the normal path. No new timer or polling: the wait uses the existing persisted `nextAttemptAt` deadline and the existing resume dispatch. - Not verified: a live provider account at its usage limit, and a live remote sandbox that waits through a full cooldown. The full repository suite was not run locally; CI ran it. ## Risks - A task that waits for a cooldown stays `in_progress` with its execution lock, for up to one day. This is the existing same-run retry state. Stop, reassignment, pause and an explicit Retry keep their current behavior through the existing claim checks. - During the wait the failed run keeps its environment lease in the same way as any same-run retry. A remote lease with an unexported prepared workspace is retained by the orphaned-lease sweep until the retry runs. Other active leases of the failed run are released after the usual five minutes, and the retry acquires a new one. - A `rate_limit` failure without a reset time (for example a short HTTP 429) delays only startup-failure retries of that agent, by up to one hour. Normal runs are not delayed, and a later success clears the evidence. - A startup failure that needs user action (for example a sign-in failure) now uses two automatic retries, about 90 seconds in total, before the task is blocked. Before, it was blocked at once with an ambiguous reason. - The agent's other queued runs are not held during a cooldown. Each one fails at `session.open` without provider work and then waits for the same cooldown. An agent-wide admission hold has no clean extension point in the current queue, so it is not part of this change. - Re-arming consumed scheduled monitors is out of scope. This change makes it matter less: a run that consumed a monitor wake now retries itself on the same run. A monitor that a run consumed before its retry budget was used stays consumed. - No schema or API change. The new error code appears on runs and in run error codes. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used Anthropic Claude Opus 5.5 (`claude-opus-5-5`) through Claude Code, with extended reasoning and tool use (shell, file editing, test execution). The host does not expose the context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-03 13:24:59 -08:00
DottaandPaperclip	f79264d253	perf(recovery): project current wait facts during action reconciliation (#18793 ) ## Thinking Path > - Paperclip manages work and execution for AI agents. > - Recovery actions keep unresolved work visible and preserve its owner. > - Action reconciliation checks whether a durable wait or another live path has appeared. > - Its current-wait check collected a complete retry fingerprint through seven queries. > - This caller used only the durable-wait flag and reason. > - This pull request reads those facts in one fresh statement and shares the existing reason policy. > - The local service uses less CPU. Fresh mutation guards and full retry fingerprints stay in place. ## Linked Issues or Issue Description What existing behavior does this improve? Active recovery action reconciliation. Subsystem affected server/ — orchestration and recovery. Current behavior Each admitted source action collects blockers, children, interactions, approvals, work products, runs and wakes. It sorts and hashes these records. This caller uses only the durable-wait flag and reason. Proposed behavior Read three current existence facts in one company-bound statement. Use the same reason policy as the complete fingerprint collector. Keep child checks, outside-action checks and mutation services. Reason and benefit The actual local PostgreSQL service uses less CPU across 8, 32 and 100-action synthetic cohorts. This removes unused work on mostly empty results. The live workload mix and fleet benefit remain unknown. Breaking changes No API or schema changes. Query facts now share one statement snapshot. Scalar issue fields still come from the source cohort read. Additional context No duplicate projection change was found in the public PR search. The roadmap has no overlapping core feature. This is a narrow performance change. ## What Changed - Add a current durable-wait reader and share reason selection with the full fingerprint collector. - Use the reader only in active action reconciliation. Keep the full collector for retry and child policy. - Add PostgreSQL regressions for reason priority, scope, board waits, native holds, healthy children and concurrent decisions. - Add a manual reproducer and local paired results. Each invocation uses a unique evidence directory and exclusive output files. ## Verification - Focused PostgreSQL regressions: 11 passed. - Separate actual-service proof: 7 passed. Four CPU tests were excluded. Results, records, semantic relation edges and notifications match the pinned baseline. Late query facts and a concurrent action decision are covered. - Retained primary local paired CPU medians: 8 actions 7.091 → 2.886 ms; 32 actions 21.922 → 8.374 ms; 100 actions 68.384 → 32.718 ms. All nine CPU and wall pairs improve at these sizes. The data file retains all 72 primary samples and ranges. - Warm application counts: 81 → 33 queries at 8 actions; 321 → 129 at 32; 1001 → 401 at 100. Rows rise from N to 2N because each EXISTS statement returns one row. - See `doc/benchmarks/2026-11-03-active-recovery-projection.md` and its JSON data for commands, provenance and limits. - `pnpm -r typecheck` and `pnpm build` passed on head `f2f8c4a755ac67679d78c6005f420a21a101c1ad`. - Shipped manual helper smoke with `--proof-only`: 7 passed, 4 timing tests skipped. All count-proof sample arrays are empty. Unique output names and cleanup of the three temporary source files were verified. - Exact-head CI completed with 52 successful checks, three intentional skips and a successful Snyk status. Greptile completed successfully at 5/5 with no inline findings. The `review` workflow passed commitperclip dependency and PR quality gates. No separate Paperclip Review check or verdict was published. The local `pnpm test:run` attempt passed all six chat shards (1,834 tests), then reported 62 Muse failures and one gateway failure in the general-server batch. The initial cases each reached the unchanged 15-second timeout. The Muse case was followed by fast failures. After the full batch breadth and complete CI coverage were established, the owned known-failed run was interrupted once with SIGINT. Remaining general-server, serialized and workspace scopes were not completed by that local invocation. Its interrupted tool exit code is not a pass. Each initial timeout case then passed in one fresh isolated control. A single fresh full Muse file passed all 256 tests. Source review found no changed projection path in either initial case. The original timeout causes remain unresolved; later passes and green CI do not erase them. The first reserved CPU run lost its raw files after a harness path error caused an unintended repeat to overwrite them. Its summary and log remain historical evidence. The separately retained repeat is the primary inspectable evidence above. It had no renewed quiet reservation. The runs are not pooled. The corrected untimed proof and public helper prevent output replacement. No third CPU run was made. ## Risks - One statement observes its query facts together. This does not reproduce every race history of seven concurrent statements. - Existing action update predicates, issue mutation locks, board ownership, native reconciliation holds and company guards remain in place. - The evidence is a local source service benchmark on macOS and real PostgreSQL. It is not deployed CPU or request latency acceptance. The census was startup-adjacent. Conditional scale suggests only a modest fleet return. - The original local batch timeout causes remain unexplained. No hosted, Sentry or OTLP improvement is claimed. No merge or deployment is requested. ## Model Used OpenAI `gpt-6.1-sol`, with reasoning, tool use and code execution. The host did not expose the context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (focused and isolated controls; the broad local failure/interruption is described above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-03 10:13:03 -06:00
DottaandPaperclip	a92f616cd0	fix(db): attribute bounded captures to execution owners (#18634 ) ## Thinking Path > - Paperclip manages agent work through shared server and database services. > - High query volume can spend main-thread time on SQL construction and result decoding. > - The existing bounded response capture counts rows and bytes, but it cannot name the operation that executed a query. > - Bundled callers can be missing, and measuring every parsed JSON value adds work to the diagnostic. > - This pull request carries a semantic execution owner into each measured query and bounds response sizing. > - Operators can rank query volume by owner before choosing a performance fix. ## Linked Issues or Issue Description Refs #17504, #17522, #17537, #17564. What existing behavior does this improve? The opt-in `PAPERCLIP_DB_RESPONSE_CAPTURE_WINDOW` diagnostic. Subsystem affected Database diagnostics and existing server execution boundaries. Current behavior The capture groups by table, operation, and builder caller. It cannot distinguish execution owners that reuse a prepared query. Bundled deployments need source maps for caller locations. The capture visits every returned value and serializes parsed JSON again to estimate its size. Proposed behavior Version 2 captures use fixed semantic owners for HTTP requests, delivery queues, execution phases, and budget work. They count completed calls, returned rows, errors, and completion wall time. They sample successful results 1, 17, 33, and so on, with fixed cell and string/byte limits. Complex values remain explicitly unmeasured. Semantic ownership works without source maps. Reason and benefit A short census can identify which operations issue the most queries or return the most rows. It preserves each query's execution owner through pooled connections, concurrent operations, and prepared-query reuse. It reduces diagnostic work without claiming per-query CPU attribution. Breaking changes Opt-in capture output moves from `entry.v1`/`summary.v1` to `entry.v2`/`summary.v2`. Sampled primitive byte lower bounds replace all-result byte estimates. Consumers must use the new fields and must not extrapolate them into a total. There is no database migration or application API change. ## What Changed - Added `withDatabaseOperation(owner, action)` to the DB package. The server owns semantic labels. Queries retain execution ownership in their completion closure. - Added fixed owners at existing HTTP, delivery, budget, and execution timing boundaries. Delivery work replaces the context of the producer that woke it. - Added complete call/row/error counters and completion wall durations within the existing bounded window. Durations include waits and are not CPU measurements. - Sampled and bounded primitive-value sizing. Complex and oversized values remain unmeasured. Known owners skip repeated stack extraction. - Preserved the deadline, group limit, paced output, query outcomes, and existing coverage exclusions. Updated the operator documentation. - Added real PostgreSQL owner-isolation and prepared-query tests, server scope tests, and plain bundled-runtime proof. - Moved projection-test and benchmark full-response sizing into an explicit test helper. Existing payload-size thresholds remain unchanged; these synthetic checks do not use the production sampler. ## Verification - Final-commit DB capture suites: 43 tests passed in 13.86 seconds against the actual PostgreSQL driver, including concurrent owners, prepared reuse, transaction failure, a dedicated client, sampling, and a query crossing the deadline. DB typecheck passed. - Server attribution and existing delivery/execution/budget suites: 53 tests passed. Final concurrent HTTP GET/JSON POST scope refinement: 5 tests passed. - Runtime bundle suite: 8 existing tests passed. The new no-source-map owner test passed after its fixture was corrected to use a driver thenable. An ordinary native Promise can bypass an overridden `.then`; the real PostgreSQL tests separately cover driver integration. - Observer-only experiment: three alternating pairs per case on Node 24.22.2, macOS arm64, 3,000 calls per run after warm-up. For an already-materialized synthetic JSON result, median capture CPU fell from 82.37 to 1.87 microseconds/call. For a 100-row primitive result, it fell from 8.10 to 4.40 microseconds/call. Every result retained identity and all calls were counted. The new diagnostic intentionally measures less byte data; these are observer costs, not database latency or a hosted application speedup. Disabled-case timings were below one microsecond/call and too small for a performance conclusion. - Final lifecycle regression: 28 unit tests and DB typecheck passed. Closing the last capture disables and replaces owner storage; overlapping captures retain it. - Final-commit full recursive typecheck passed in 38.38 seconds and full build passed in 40.81 seconds. The build stamp matches `83ed4ba9cebd68d48929c46f15ab32d5c0b9687b`. Focused server attribution tests also passed 5/5 before the test-helper-only migration. Full test acceptance will use the repository's parallel CI gates; the unsharded local command takes hours. - Verification failures retained: initial sandbox runs could not open local IPC/PostgreSQL listeners; permitted reruns exercised the real services. One old test expected JSON serialization and was updated to assert explicit unmeasured complex values. The first bundle fixture used the wrong promise boundary, as described above. A retained-callback regression also demonstrated that disabling and reusing the same AsyncLocalStorage instance could resurrect an old owner; replacing the instance after the last capture closes fixes that case. - The first CI run found older projection tests consuming the previous full-response fields. Those consumers now use the full-response test helper, preserving their byte thresholds. The two complete affected route suites plus helper tests passed 115/115 in 73.08 seconds. A temporary full-row query negative control then failed the unchanged 2,048-byte limit at 5,000,812 bytes and was restored; five other service suites passed 34/34 and the affected chat cases passed 3/3. All three previously failing CI shards passed on the corrected head. A local skill-list benchmark smoke check completed 255 captured queries with no in-flight queries or overflow; this was integration verification under concurrent test load, not a timing benchmark. - Two final-head runner jobs lost communication with GitHub before reporting a test result. One targeted retry of those jobs and their dependent aggregate passed. All 53 final-head checks passed, with 3 intentional skips; previously passing suites were preserved. - Hosted capture remains pending until the reviewed revision is deployed and a bounded window is explicitly configured. Reproduce the focused checks with Node 24 and the repository pnpm version: ```sh pnpm exec vitest run packages/db/src/response-capture.test.ts packages/db/src/response-capture-postgres.test.ts pnpm --dir server exec vitest run --config vitest.config.ts src/__tests__/database-operation-attribution.test.ts scripts/bundle-runtime.test.mjs pnpm -r typecheck pnpm build ``` ## Risks - Owner context is active while a valid future capture window is armed, up to 30 minutes before accounting starts. No-window and expired calls take the direct path. Diagnostics still add bounded work while enabled. - A nearest nested owner is reported. Broad `http.request` and unattributed groups remain visible; this does not identify every service callsite or SQL shape. - The existing census excludes tagged/raw explicit-execute/streaming paths and internal protocol commands. Its counts are not PostgreSQL transaction totals. - Completion wall time includes queue/database/network/scheduling waits. Concurrent durations overlap. Sampled bytes exclude complex values, large strings, and values beyond the sizing limits. - Activation is startup-only. Operators must choose the window after artifact preparation, allow rollout/warm-up time, and remove the expired setting afterward. ## Model Used OpenAI Codex, GPT-6 family, with code execution, repository inspection, and parallel agents. The exact deployment model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-02 12:12:21 -06:00
DottaandPaperclip	9ca9af544a	Drive remaining live run and resource recovery from durable work (#18419 ) Reviewed and verified at `6a4c193906`. Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-10-31 16:00:35 -05:00
DottaandPaperclip	42fd568cf8	Drive issue recovery from committed changes and exact deadlines (#18402 ) Reviewed and verified at `11b49125db`. Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-10-31 09:41:27 -05:00
DottaandPaperclip	baf651c8d0	perf(heartbeat): drive retry promotion from durable deadlines (#18387 ) Reviewed and verified at `f1c48c924d`. Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-10-31 09:07:52 -05:00

1 2 3 4