6 Commits

Include renames

Author	SHA1	Message	Date
3 people	bbcf5dfe98	fix(heartbeat): renew legacy controller leases on a private pool (#17032) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A legacy (non-native) agent run keeps a controller lease on its `heartbeat_runs` row. The server renews the lease every 10 seconds. If no renewal succeeds in 60 seconds, the watchdog stops the run with `renewal_deadline`. > - #16359 added timings to lease-loss reports. The first staging event with these timings shows `deadlineLateMs=0`, `renewalPendingMs=50000`, and 108 renewals in about 22.5 minutes, where about 135 are expected. Thus the event loop was not blocked. One renewal UPDATE was in flight for 50 seconds, and renewals were slow for a long time before the loss. > - The agent had exited with code 0. The run was in post-execution `sync_back`. Other runs on the same host completed. No code path held a lock on the run row across slow work. A later `instruction_save` UPDATE on the same row completed while the earlier renewal was still pending, so a row-lock wait does not explain the stall. > - postgres.js 3.4.9 sends a new query to a busy connection when no connection is idle (`handler` → `go(busy.shift(), query)` in `src/index.js`). The query waits behind the statement that is in flight on that connection. The renewal had no timeout, and the `pending` flag stopped all later renewals. Thus one slow unrelated statement on the shared pool can stop a healthy run. > - This pull request moves lease renewal to a small private pool, bounds each attempt with a statement timeout, and retries a timed-out attempt until the existing lease deadline. > - The benefit is that slow application queries on the shared pool can no longer cancel a healthy legacy run. ## Linked Issues or Issue Description Refs #16359 (lease-loss timings that identified this cause) Refs #16358 (pre-dispatch lease loss becomes retryable; different path) Refs #16917 (open: keeps a clean adapter result after a lease lapsed; it does not prevent the loss) Refs #13389 (open: host sleep / late timers; different cause) **What happened?** A staging legacy `claude_local` run was cancelled with `Legacy controller lease lost (renewal_deadline)` after the agent exited with code 0. The timings show that the deadline timer ran on time, and one renewal UPDATE was pending for 50 seconds. **Expected behavior** A live controller keeps its lease. Slow unrelated database work does not cancel the run. **Steps to reproduce** 1. Create a database client with one pooled connection. 2. Start `select pg_sleep(5)` on that client. 3. Call `watchLegacyControllerLease(...).assertOwned()` with the same client. 4. Before this change, the renewal waits about 5 seconds behind `pg_sleep`. With a long enough slow statement, the lease deadline stops the run. ## What Changed - `packages/db`: Add `livenessDb(db)`. It gives each `createDb` client a private pool of at most 2 connections (`application_name=paperclip-controller-lease`). The pool is created on first use. Test doubles and transactions get the original client back. - `packages/db`: Add `closeLivenessDb(db, timeoutSeconds)`. The server calls it in `closeDatabaseClients`, so shutdown and failed startup end the private pool too. - `legacy-controller-lease.ts`: The watchdog renews in a transaction on the private pool. Each attempt sets a transaction-local `statement_timeout` of 10 seconds. A timed-out attempt (`57014`) is retried. After a successful retry, the local deadline is measured from that attempt's start, because that attempt's UPDATE wrote the fresh 60 second lease. The existing 60 second deadline, the abort race, and the `renewal_rejected` / `renewal_error` attribution do not change. `renewLegacyControllerLease` itself does not change, so the runtime-selection caller inside its own transaction is not affected. - Tests: a saturated shared pool no longer delays renewal; a renewal that times out on a locked run row retries and keeps the lease; the private pool is cached per client, has its own `application_name`, and closes; after a 45 second timed-out attempt and a successful retry, the next deadline is a full 60 seconds after the retry. The existing test doubles now model the transaction. - `docs/deploy/database.md`: Count the up to 2 extra connections per server process when you size `max_connections`. ## Verification - `cd server && npx vitest run src/services/legacy-controller-lease.test.ts`: 20 passed, on embedded Postgres. - Red/green: with the old `legacy-controller-lease.ts` and the new tests, the saturated-pool test fails with `expected 5018 to be less than 2000`. With the fix, it passes. - Red/green for the retry deadline: with the deadline measured from the first attempt, the retry-deadline test fails (`expected true to be false`) because the run is aborted 15 seconds after the retry. With the fix, it passes. - Mutation check: when the timeout retry is removed, the locked-row test fails with `canceling statement due to statement timeout`. - `npx vitest run src/__tests__/heartbeat-process-recovery.test.ts -t "lease|dispatch|controller|interaction"`: 43 passed. - `npx vitest run src/services/heartbeat/run-completion.test.ts src/__tests__/heartbeat-orphaned-active-lease-sweep.test.ts`: 53 passed. - `npx vitest run src/__tests__/server-startup-feedback-export.test.ts`: passed. - `cd packages/db && npx vitest run src/work-signals.test.ts src/client-options.test.ts src/client-teardown-registry.test.ts`: 24 passed. `npx tsc --noEmit` for `packages/db` passed. - Not run locally: the full server `tsc` and the full server suite. The sandbox has a 4 GB memory limit, and `tsc` was killed (exit 137). CI is the gate for the server typecheck, the full tests, and the build. ## Risks - Each server process can open up to 2 more Postgres connections. Deployments that are at their `max_connections` limit must allow for them. The docs now say this. - A renewal attempt is now 4 statements (BEGIN, `set_config`, UPDATE, COMMIT) instead of 1. Each statement is a short primary-key operation on a dedicated connection. - If many runs wait on locked run rows at the same time, renewals for other runs can queue on the 2 private connections. The 10 second statement timeout limits each wait, which is less than the 60 second lease. - Behavior for a real lease loss does not change. A rejected renewal and a non-timeout error still stop the run at once, and the 60 second deadline still applies. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code in agent mode with tool use (shell, file edit, and test execution). A read-only subagent helped with code search. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-20 18:13:53 -07:00
3 people	76164b53cb	fix(heartbeat): retry a legacy run that lost its lease before dispatch (#16358) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The heartbeat runner executes legacy agent runs. A controller lease watchdog keeps durable ownership of each legacy run, and it aborts the run when it loses the lease. > - #16084 records a lease loss as a `controller_lease` cancellation when the adapter returns or throws. It does not cover a lease loss during setup, before the adapter starts. > - In setup, the abort reason goes to the setup catch. The catch records an opaque `setup_failed` and sends the raw `LegacyControllerLeaseLostError` to Sentry with no cause label. > - Issue continuation recovery treats `setup_failed` as non-retryable. The task moves to `blocked`, but no provider work started. > - This pull request records a lease loss before dispatch as `controller_lease_lost`, with the same `controller_lease` evidence. Recovery retries it as transient infrastructure, and so does the accepted-interaction (plan approval) continuation retry. > - The benefit is that a host lease loss during setup no longer strands the task, and the next Sentry event shows which renewal failure caused the loss. ## Linked Issues or Issue Description Refs #16084 Refs #16355 Refs #16359 **What happened?** A staging server reported `LegacyControllerLeaseLostError: Legacy controller lease lost` from the heartbeat. The run failed during setup with `setup_failed`. Continuation recovery then said "non-retryable failure on this issue's continuation run (`setup_failed`)" and moved the task to `blocked`. A scanner had to move the task back to `in_progress` by hand. **Expected behavior** A lease loss before adapter dispatch is a host infrastructure stop. The run records the `controller_lease` cause, like the adapter-phase path. The run keeps `providerWorkStarted: false`. Recovery retries the continuation with its bounded transient budget, and it does not block the task. **Steps to reproduce** 1. Start a legacy run that has a controller lease. 2. Before the adapter dispatch, make the lease renewal fail: the update matches no row, the query errors, or the lease deadline passes. 3. `controllerLease.assertOwned()` throws the abort reason in workspace preparation or before dispatch. 4. Before this change: the run is `failed`/`setup_failed`, `resultJson` has no `cancellation`, and continuation recovery blocks the task. After this change: the run is `failed`/`controller_lease_lost` with `cancellation.source: "controller_lease"` and a closed cause, and continuation recovery classifies it as `transient_infra`. **Additional context** #16355 handles a different path: a restart orphan that the next boot reaps as `process_lost`. This change handles a live controller that loses its own lease during setup. This change does not find the root cause of the lease loss. The closed cause label in the next Sentry event gives that fact. ## What Changed - `server/src/services/heartbeat/run-completion.ts`: `failRunSetup` receives the execution abort signal. When the abort reason is a `LegacyControllerLeaseLostError`, it writes error code `controller_lease_lost`, adds the `controller_lease` cancellation evidence, and keeps the existing bootstrap recovery facts. The agent goes back to idle, not `error`, because the stop is a host fault. This applies only before provider dispatch: `failRunSetup` also receives `providerDispatchStarted`, and a setup-path error after a legacy adapter or native dispatch began keeps `setup_failed`, so started work is never retried. - `server/src/services/legacy-controller-lease.ts`: add the `CONTROLLER_LEASE_LOST_FAILURE_CODE` constant. - Merged with #16359 (lease-loss timings): the setup failure keeps its `resultJson.controllerLeaseLoss` timings and adds this PR's `controller_lease` cancellation from the abort signal. #16359's setup test now aborts the signal first, as `assertOwned()` does, and expects `controller_lease_lost`. - `server/src/services/heartbeat.ts`: pass the execution abort signal and `providerDispatchStarted` (`legacyAdapterEntered || nativeDispatchStarted`) to `failRunSetup`. - `server/src/services/recovery/service.ts`: add `controller_lease_lost` to the transient-infrastructure continuation codes (3 bounded attempts with backoff). - `server/src/services/heartbeat/retries.ts`: add `controller_lease_lost` to the accepted-interaction continuation retry allowlist, next to `process_lost`. Without it, a plan-approval continuation that lost its lease before dispatch was escalated and blocked before generic recovery could see it. - `server/src/services/heartbeat/run-lifecycle.ts`: add `controller_lease_lost` to the pre-adapter setup codes, so the comment policy does not queue a missing-comment retry. - `doc/run-log-events.md`: document the new error code. - Tests: setup lease loss when the abort reason is thrown, and when a setup step throws a different error after the abort. A unit test shows a lease loss after dispatch keeps `setup_failed`. A classifier test shows `controller_lease_lost` is transient and `setup_failed` stays non-retryable. Two assembled heartbeat tests (`heartbeat-process-recovery.test.ts`) lose the real lease during workspace preparation: one for a plain issue continuation (one `transient_failure` retry) and one for an accepted plan-approval interaction continuation (one `interaction_continuation_infra_retry` retry). Both check the saved run, no adapter dispatch, and the issue still `in_progress`. ## Verification - `cd server && npx vitest run src/services/heartbeat/run-completion.test.ts`: 25 passed (embedded Postgres), including the new post-dispatch test. - `cd server && npx vitest run src/services/run-cancellation.test.ts src/services/recovery/provider-failure-classification.test.ts`: passed on the first head (29 with run-completion; 25 classifier). - `cd server && npx vitest run src/services/legacy-controller-lease.test.ts`: 13 passed (needs `pnpm --filter @paperclipai/plugin-sdk... build` first). - After merging master (head `2148b27`): `run-completion`, `legacy-controller-lease`, `run-cancellation`, `provider-failure-classification` and `run-failure-report` suites, 310 passed; `heartbeat-process-recovery.test.ts -t "lease|dispatch"`, 31 passed. - Head `c52825c`: `retries`, `run-completion`, `heartbeat-retry-scheduling`, `legacy-controller-lease` and `provider-failure-classification` suites, 134 passed; `heartbeat-process-recovery.test.ts -t "lease|dispatch|interaction"`, 41 passed. - Negative controls: with the lease-loss branch in `run-completion.ts` disabled, the plain-continuation assembled test fails (`setup_failed` instead of `controller_lease_lost`). With the `retries.ts` allowlist change reverted, the accepted-interaction assembled test fails because no retry is scheduled. - Not run locally: full `tsc` and `pnpm build`. The sandbox has a 4 GB memory limit and the OOM killer stops `tsc`. CI is the typecheck and build gate. ## Risks - Low risk. The change applies only when the execution abort reason is a `LegacyControllerLeaseLostError` in the setup catch before dispatch. Other setup failures keep their current codes. - The retry uses the existing continuation recovery budgets and admission checks (agent state, budget, pause). The interaction-continuation retry keeps its existing 2-attempt budget. The retry is a new run. The revoked controller cannot dispatch, because its renewal compare-and-set fails. - A run that the reaper settles first keeps the reaper outcome. `setRunStatusIfRunning` makes the first terminal write win. - Sentry still gets this unexpected stop, with the closed `controllerLeaseLossCause` label. If the lease loss is frequent, the event volume does not change. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass (targeted suites pass; typecheck/build not possible in this sandbox, see Verification) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Devin Foley <139239+devinfoley@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-14 21:31:42 -07:00
3 people	ccc77c5b5e	fix(server): record legacy controller lease-loss timings (#16359) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The heartbeat runner executes legacy agent runs, and a controller lease watchdog keeps durable ownership of each run > - The watchdog renews the lease every 10 seconds and stops the run when a renewal does not finish within the 60-second lease > - A staging run failed in setup with `LegacyControllerLeaseLostError` and the `renewal_deadline` cause, after about eight minutes of preparation and with no server restart > - The setup failure path did not record the cause, and no evidence showed why the renewal did not finish: a stalled database query or a blocked event loop > - This pull request records bounded watchdog timings with each lease loss, on the setup failure path and on the cancellation path > - The benefit is that the next event shows which of the two causes stops renewal, which is the missing fact to fix the root cause ## Linked Issues or Issue Description Refs #16084 (added the closed lease-loss cause for cancelled runs) Refs #16355 (retries a run that a restart stopped before dispatch; a different path) **What happened?** A legacy `paperclip_runner` run on staging failed with `setup_failed` and the message "Legacy controller lease lost". The stack shows the watchdog deadline timer (`renewal_deadline`). The run was in `executionStage: "preparing"` for about 467 seconds. The server started about 11 minutes before the run, so this was not a restart. The run record and the Sentry event do not show the lease-loss cause, because the setup failure path does not read it. Nothing shows if a renewal query was in flight at the deadline, or if the deadline callback ran late. **Expected behavior** When the watchdog loses the lease, the run records the closed cause and the watchdog timings. This applies when the loss fails setup and when it stops the adapter. The failure report contains only the closed cause and bounded numbers. **Steps to reproduce** 1. Start a legacy run with a controller lease. 2. During setup, make a lease renewal query hang for more than 60 seconds, or block the event loop past the deadline. 3. Let the watchdog abort the run. 4. Read `resultJson` on the failed run. Before this change, it has no lease-loss evidence. ## What Changed - `legacy-controller-lease.ts`: The watchdog tracks the last successful renewal, the renewals in flight, the scheduled deadline, and the renewal count. `LegacyControllerLeaseLostError` now carries `leaseLossTiming`: `sinceRenewalMs`, `renewalPendingMs` (or `null`), `deadlineLateMs` (only for `renewal_deadline`), and `renewalCount`. Ownership behavior does not change. - `run-cancellation.ts`: Add the `LegacyControllerLeaseLossTiming` type and `readControllerLeaseLoss()`. The reader accepts only a closed cause and non-negative finite numbers, and clamps each value to 24 hours. - `heartbeat/run-completion.ts`: Persist `resultJson.controllerLeaseLoss` with the cause and timings. This applies to a lease loss that stops the adapter and to a lease loss that fails setup (the thrown abort reason). - `run-failure-diagnostics.ts`: Report `controllerLeaseLossCause` and the four bounded timing fields from `controllerLeaseLoss`. A setup failure now gets the cause label too. - `doc/run-log-events.md`: Document the new record and how to read the timings. - Tests: watchdog timings for a stalled query, a blocked event loop, and a failing query; completion records the evidence for an adapter stop and for a setup failure; the failure report clamps values, drops unknown keys, and drops a record without a closed cause. ## Verification - `pnpm --filter @paperclipai/server exec vitest run src/services/legacy-controller-lease.test.ts src/services/heartbeat/run-completion.test.ts src/services/__tests__/run-failure-report.test.ts src/services/run-cancellation.test.ts` - I could not run these tests locally. The sandbox has a 4 GB memory limit, and `pnpm install` was stopped by the out-of-memory killer during lockfile resolution. CI is the test run for this change. - After deployment, a new lease loss shows `controllerLeaseRenewalPendingMs` and `controllerLeaseDeadlineLateMs` in the Sentry execution context. A pending time near 50 seconds or more points at the database. A large deadline lateness points at the event loop. ## Risks - Low risk. This change adds diagnostics. It does not change when the watchdog stops a run, and it does not prevent the lease loss. - The new `resultJson.controllerLeaseLoss` field is server-written. Readers ignore unknown values. Old rows do not have it. - The failure report gets five closed or numeric fields. It gets no reason text, IDs, or user content. - The event-loop test blocks the loop for about 80 ms to make the deadline late. It uses real timers, so a very slow CI host makes the lateness larger, not smaller. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, Sentry and GitHub connectors). The exact context-window size is not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass (not possible in this sandbox; see Verification) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Devin Foley <139239+devinfoley@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-14 19:52:23 -07:00
Devin FoleyandClaude Opus 5.5	5e8956c207	fix(heartbeat): retry legacy runs a restart stopped before dispatch (#16355) ... A hard server restart can stop a legacy run after it is claimed but before the adapter is dispatched. The next boot reaped it as failed/process_lost with no retry and held it for operator reconciliation, so the task wake was lost. The legacy controller only writes executionStage "dispatching" in the lease renewal that admits the adapter. The orphan reaper now revokes the expired foreign-boot controller first, and uses the stage and process identifiers returned by that same UPDATE ... RETURNING. If the run is still "preparing" with no process identifiers, it records bootstrap evidence (providerWorkStarted: false) and uses the existing one-time process-loss retry. In "unexpected" mode, run-failure reporting does not send this retried restart interruption to Sentry. Runs that reached dispatching, native runs, and runs owned by the current boot are unchanged. Verified exact head 91f1352097: full CI green, Greptile 5/5, no unresolved review threads, no conflicts. The tests were not run locally (sandbox memory limit); CI is the test run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-14 18:27:11 -07:00
4 people	a64969b288	fix(server): attribute legacy runs stopped by controller lease loss (#16084) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The heartbeat runner executes legacy agent runs. A controller lease watchdog keeps durable ownership of each legacy run. > - When the watchdog loses the lease, it aborts the adapter. The abort reason was a plain `Error`, and nothing recorded why the run stopped. > - Completion then saved the cancellation as source `unknown`. Sentry got an unexpected cancel that it could not tell apart from other unrecorded stops. > - We saw this on a sandboxed Grok run. The system cancelled it after about 22 minutes with `cancellationSource: unknown`. Workspace copy-back then failed. Every `cancelRun` path records a stop request, so an unrecorded stop this late points to the lease watchdog. > - This pull request records a lease loss as its own cancellation source, with a closed cause label. > - The benefit is that the next event shows whether the lease loss caused the stop, and which renewal failure caused it. That is the missing fact to fix the root cause. ## Linked Issues or Issue Description Refs #15790 Refs #15920 **What happened?** A legacy sandbox run was stopped by the system while it was working. The run record and the Sentry diagnostics show `cancellationSource: unknown`, with no initiator. The controller lease watchdog aborts the adapter with a plain `Error("Legacy controller lease lost")`. Completion does not read the abort reason, so it records the stop as `unknown`. **Expected behavior** When the host stops its own adapter because it lost the controller lease, the run records that cause. Diagnostics name the cause with a closed label. A stop that a user or the control plane requested earlier keeps its recorded source. **Steps to reproduce** 1. Start a legacy run that has a controller lease. 2. Make lease renewal fail: the renewal query errors, the update matches no row, or the lease deadline passes. 3. Let the adapter return its cancelled result. 4. Read `resultJson.cancellation` on the run. Before this change, `source` is `unknown`. **Agent adapter(s) involved** Not adapter-specific (core bug). We saw it with `grok_local` in a sandbox. **Additional context** This change adds diagnostics. It does not prevent the lease loss. #15920 retries a transient copy-back failure. It does not help here, because the stopped sandbox rejects every retry. ## What Changed - `legacy-controller-lease.ts`: The watchdog aborts with a new `LegacyControllerLeaseLostError`. The error carries a closed cause: `renewal_rejected`, `renewal_deadline`, or `renewal_error`. - `run-cancellation.ts`: Add the `controller_lease` cancellation source. Add `controllerLeaseLossCancellation()` and `readControllerLeaseLossCause()`. The reader accepts only the server-written reason text. - `heartbeat/run-completion.ts`: When the outcome is `cancelled` and the abort reason is a lease loss, record a `controller_lease` cancellation (`expected: false`). This covers both an adapter that returns a cancelled result (`completeRun`) and one that throws after the abort, for example through `signal.throwIfAborted()` (`failRun`). A stop request recorded earlier still takes precedence. - `run-failure-diagnostics.ts`: Add `controllerLeaseLossCause` to the execution diagnostics. Only the closed label goes to Sentry, never the reason text. - `heartbeat/run-state.ts`: preserve `controller_lease` in the bounded SQL projection when saved results exceed 64 KB. A database regression verifies the default run-detail read. - `canContinueCancelledRun` stays unchanged. It does not accept `controller_lease`, so a lease loss does not authorize automatic continuation. - `doc/run-log-events.md`: Document the new source and cause labels. - Tests: unit tests for the cause on each watchdog path, completion tests that record the source when the adapter returns and when it throws the abort reason (both fail without the change), tests that a recorded stop request takes precedence, and a test that the Sentry report contains the closed label and not the reason text. ## Verification - `cd server && npx vitest run src/services/run-cancellation.test.ts src/services/legacy-controller-lease.test.ts src/services/__tests__/run-failure-report.test.ts src/services/heartbeat/run-completion.test.ts`: 238 passed, including the embedded Postgres suites. - I reverted only `run-completion.ts` and ran the new completion test. It failed with `source: "unknown"`. With the change, it records `controller_lease`. - I removed only the `failRun` attribution and ran the new thrown-error test. It failed without the `controller_lease` record, and passes with it. - Final full repository `pnpm -r typecheck` and `pnpm build` pass. - Final local regression verification: all 247 tests pass across run-state, cancellation, watchdog, failure reporting, and completion suites. The large-result test fails before the SQL allowlist change. All latest-head CI checks pass (54 successes and two intentional skips). Fresh Apex is 5/5 on `708856c568d07d7b446a4a4af2edc6d9eec7d155` with no unresolved findings. ## Risks - Low risk. The change adds a new source value to server-owned JSON. Old rows keep their values. The readers accept the new value, and no UI code switches on this field. - A lease-loss cancel stays `expected: false`, so it is still reported as unexpected, as before. Now it has a more specific source. - The Sentry payload gets one new closed label (`controllerLeaseLossCause`). It does not include reason text or IDs. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, test runs). OpenAI GPT-6 (`gpt-6`) through Codex added the large-result regression and SQL projection fix, with shell, file-edit, and test tools. The exact context-window size is not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-13 14:50:40 -07:00
Nicky LeachandPaperclip	dbf5ea432d	fix: protect starting runs during overlapping deployments (#13285) ... ## Thinking Path > - Paperclip controls agent work across service deployments. > - A run can provision a remote sandbox before a process or invocation event exists. > - Each container previously treated its own missing process handle as proof that the run was orphaned. > - Overlapping deployments could therefore fail a run owned by another container. > - This pull request records and renews a controller lease before provisioning. > - A recovery worker must revoke an expired owner before it finalizes the run. ## Linked Issues or Issue Description Merged PR #13272 records startup adapter identity and restores explicit user continuation. This PR adds controller ownership on top of current master. Refs #7997 and #10442 for related replica and ownership problems. Related #13138 addresses silence and detached local processes; this change does not infer death from silence. **What happened?** During an overlapping hosted service deployment, a new container reaped a legacy conversation run that another container was provisioning. The run had no PID or adapter invocation yet. **Expected behavior** A live controller keeps its run. After controller loss, one recovery worker takes cleanup authority and the old controller cannot dispatch further work. **Steps to reproduce** Claim a legacy run in controller A. Start controller B against the same database before A finishes provisioning. Run the startup reaper in B. **Paperclip version or commit** Observed on `663c44cb2b9c28336d38d0b4a6971f4f1964bce6` in a hosted Railway deployment with a Daytona environment. ## What Changed - Add nullable controller boot ID, lease deadline, and execution stage columns. Claim ownership in the queued-to-running update. - Renew ownership independently of run output. Abort and reject dispatch if renewal fails. - Serialize reaper revocation against renewal. Let unfinished recovery claims expire after a restart. - Restrict graceful shutdown to legacy runs owned by the current controller. - Hand ownership back to the existing native coordinator when runtime selection becomes native. - Add twelve database regressions and document the lease contract. Update the task-drain regression to require controller expiry before reaping. ## Verification - `pnpm exec vitest run server/src/services/legacy-controller-lease.test.ts server/src/__tests__/heartbeat-task-drain-admission-release.test.ts`: 14 passed after rebasing onto master (`f12b647ae`). - Queue-interruption regressions in `heartbeat-process-recovery.test.ts`: 2 passed after preserving the new cleanup promotion from #13275. - `pnpm --filter @paperclipai/server exec tsc --noEmit`: passed after rebuilding runner TypeScript outputs for the updated master. Broad local tests are omitted at the maintainer’s request; CI owns broad coverage. - Latest-head CI passed on `f255e8e4d5ab2b24b12638a02434e6aa8a2285c5`: [run 34658248569](https://github.com/paperclipai/paperclip/actions/runs/34658248569). All test shards, browser suites, typecheck, build, canary, and security checks passed. Greptile is 5/5 with no unresolved review threads. ## Risks - Additive, idempotent migration; historical rows retain the previous recovery behavior. - Database unavailability aborts new dispatch rather than permitting an unfenced controller to continue. - Lease expiry is permission to clean up, not evidence that remote inference stopped. Follow-up PRs add persistent cleanup and automatic continuation. - Mixed-version deployment still includes old binaries whose reapers do not understand controller leases. ## Model Used OpenAI GPT-6 through Codex, using reasoning, repository inspection, code execution, and test tools. The precise backend revision and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-09-11 16:55:44 -07:00