Commit Graph
8 Commits
Author SHA1 Message Date
DottaandClaude Opus 5.5 576ba29fff feat(recovery): settle the legacy controller lease on the shared ownership path (#19953)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Legacy (non-native) runs hold a controller lease on `heartbeat_runs`
while the server prepares and dispatches the adapter. The shared
ownership reconciler now settles proven-gone owners (Phase 2 of the
durable ownership plan)
> - The lease stamped only a boot id. The reconciler could never prove
its owner gone (`missing_identity` in the Phase 2 data), and recovery
released it by age: lease expiry (60 s), then a 5-minute age guard on
dispatched runs
> - A hard-killed server's legacy run therefore stayed `running` for up
to 6 minutes after the next boot
> - This pull request records the controller process with the claim,
registers a compare-and-swap settle, and makes live-run recovery use the
shared prover instead of the age guard when the controller is proven
gone
> - The benefit is that a dead controller's run is settled at once,
while a live, unprovable, or previous-release controller keeps today's
behavior

## Linked Issues or Issue Description

Refs #19933 (the settle hook, merged). Stacked on #19950 (migration
0446, registry) and targets its branch until it merges; then it
retargets to master.

**What happened?**
After a hard kill, a dispatched legacy run stayed `running` until its
controller lease expired and its last update was 5 minutes old.

**What did you expect?**
The next boot proves the old controller gone and fails the run as a lost
process at once.

**Steps to reproduce**
1. Start a legacy (non-native) run and let it dispatch the adapter.
2. Kill the server (SIGKILL).
3. Start the server. The run stays `running` for up to 6 minutes.

## What Changed

- `heartbeat_runs.controller_process` (jsonb, migration 0447): `{bootId,
pid, hostHash, processStartedAt}` of the claiming server process. It
names the controller only while its `bootId` equals
`controller_boot_id`. Revocation swaps the boot id, so a revoked lease
never keeps it.
- `legacyControllerClaim` writes it in the same UPDATE (from
`currentControllerProcessStamp()`, no I/O).
- `legacy-controller-lease.ts`: `legacyControllerOwnerStamp`,
`legacyControllerProvenGone` (calls `proveOwnerGone`), and
`settleLegacyControllerLease` (compare-and-swap expiry of the proven
controller's outstanding lease after it wakes live-run recovery).
- `ownership-registry.ts`: `legacy_controller_lease` registers `settle`
and `listForRuns`. The owner is the bound process; other rows are
boot-id fences. The listing covers outstanding leases only.
- `heartbeat/recovery.ts` (3 small hunks): a controller of another boot
that the shared prover proves gone has its live lease settled through
`settleOwnableKind` and skips the age guard. The local child-process
check after it is unchanged, so a live child still keeps the run
running.
- Migration 0447 builds the partial index with `CREATE INDEX
CONCURRENTLY` (heartbeat_runs is a known-large table) and is added to
`MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS`.

Bespoke code deleted: none. The kind had no liveness proof; the age
guard stays for controllers that are not proven gone.

## Verification

- `cd server && npx vitest run
src/services/legacy-controller-lease.test.ts`: 33 passed.
- `cd server && npx vitest run
src/__tests__/ownership-reconciliation-report.test.ts`: 17 passed.
- Red with `recovery.ts` and `ownership-registry.ts` from the base
commit (tests, schema and lease module from this branch): 4 failed, 29
passed. Red: the crash-point reaper test, the expired-lease age-guard
test, the reconciler settle test, the exact-controller CAS test. The
tests were written after the code.
- New tests:
  - the claim stamps this process, and a changed boot id unbinds it;
- crash point: a controller dispatched the adapter and was killed with
its lease live; the startup reaper (default 5-minute age guard) fails
the run as `process_lost` at once;
  - a dead controller's expired lease skips the age guard;
- counter-cases keep the run and lease through the reaper and a
reconciler pass: a live earlier-boot PID, a dead PID in another PID
space, and a previous-release row with no `controller_process`; an
expired lease of a live controller stays behind the age guard;
- a read-only pass (`record: false`) never settles; a recording pass
expires the lease and leaves the run to live-run recovery;
  - settle refuses a controller that changed after the proof.
- Not run locally: typecheck (CI runs it) and the wide suites.

## Risks

- Previous-release story (rules/upgrade-safe-run-state.mdc): rows saved
by the previous release have no `controller_process`. They stay boot-id
fences with today's release (expiry, revocation, age guard). A rollback
ignores the column.
- Unprovable controllers (another host or PID space, unknown space, a
live PID with an unreadable start time) never release early. Their
handoff stays the lease expiry plus revocation, and the age guard. A
different host is never proof.
- Behavior change: a dispatched run whose controller is proven gone is
failed as `process_lost` without the 5-minute wait. The retry decision
is unchanged.
- The settle expires the lease only. Live-run recovery still settles the
run under its own checks. Accounting and workspace obligations are not
touched.
- The index build is concurrent. A cancelled build is retried by the
migrator's invalid-index handling.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, local test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 23:27:54 -06:00
Devin FoleyandClaude Opus 5.5 6a45ea479e fix(heartbeat): retry failed legacy lease renewals until the deadline (#17851)
<!-- entire-trail-link-start -->
https://entire.com/gh/paperclipai/-/trails/52/changes/paperclip/52
<!-- entire-trail-link-end -->

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A legacy agent run keeps a durable controller lease. The server
renews the lease every 10 s. The lease lasts 60 s.
> - The lease watchdog stops the run when it loses the lease, because
another controller can then take the run.
> - #17032 moved renewal to a private pool. It also retries a renewal
that hits its statement timeout. But a renewal that fails with any other
error still stops the run at once.
> - Staging still shows this failure after #17032. A legacy run made 186
good renewals over 31 minutes. Then one renewal failed with an error
23.9 s after the last good renewal, and the watchdog stopped the run
immediately (`controllerLeaseLossCause=renewal_error`). The lease still
had about 36 s left.
> - This pull request retries a failed renewal until the lease deadline,
the same as a timed-out renewal.
> - The benefit is that one dropped database connection no longer
cancels a healthy run. The 60 s lease deadline still bounds the retries,
so the ownership guarantee does not change.

## Linked Issues or Issue Description

Refs #17032
Refs #16084

**What happened?**
A legacy agent run on staging was cancelled mid-turn. The run was
healthy. The cancellation evidence was
`cancellationSource=controller_lease` and
`controllerLeaseLossCause=renewal_error`, with
`controllerLeaseRenewalCount=186` and
`controllerLeaseSinceRenewalMs=23913`. The release already contained
#17032.

**Expected behavior**
One failed lease renewal must not stop a run while the lease is still
valid. The watchdog must try the renewal again. It must stop the run
only when the lease deadline passes or when the database rejects the
renewal.

**Steps to reproduce**
1. Start a legacy run so that `watchLegacyControllerLease` watches it.
2. Make one renewal query fail with a connection error, for example
terminate its Postgres backend with `pg_terminate_backend`.
3. See that the watchdog aborts the run immediately with
`renewal_error`, although the lease has not expired.

## What Changed

- `watchLegacyControllerLease` retries a renewal that fails with an
error. It waits 1 s between attempts
(`LEGACY_CONTROLLER_RENEWAL_RETRY_DELAY_MS`). The existing lease
deadline stops the retries.
- A retried renewal is safe. The UPDATE is idempotent, and it still
requires this controller's boot ID and an unexpired lease. A rejected
renewal (no row updated) still stops the run at once with
`renewal_rejected`.
- If the deadline passes while the attempts still fail with an error,
the cause stays `renewal_error`. A statement timeout counts as a stall,
so it stays `renewal_deadline`, as before. A good renewal clears the
error state.
- `deadlineLateMs` is now recorded whenever the deadline ends the lease,
also for `renewal_error`.
- Tests:
- New embedded-Postgres test: it terminates the renewal's own backend
while the renewal waits on the run row. The renewal retries on a new
connection and keeps the lease. Without the fix, `assertOwned` rejects.
- New unit tests: one failed attempt is retried and the lease is kept; a
stall after a recovered error is still `renewal_deadline`.
- The existing `renewal_error` test now checks that a renewal that
always fails is retried and is reported as `renewal_error` at the
deadline, with its timings.
- `doc/run-log-events.md` describes the retry and the updated
`renewal_error` and `deadlineLateMs` meaning.

## Verification

- `server/src/services/legacy-controller-lease.test.ts` with embedded
Postgres: 23 tests pass.
- With the original `legacy-controller-lease.ts`, 4 of these tests fail.
One of them is the real-Postgres test that terminates the renewal
connection.
- `server/src/services/heartbeat/run-completion.test.ts` and the
`run-cancellation` tests: 35 tests pass.
- A typecheck of the changed files reports 0 errors. CI runs the full
typecheck.

## Risks

- Low risk. The change is limited to the legacy lease watchdog.
- A run with a permanently failing renewal now stops at the lease
deadline (at most 60 s after the last good renewal), not at the first
error. Another controller cannot take the run before the lease expires
in the database, so ownership stays safe.
- A non-transient error (for example a code bug in the query) is retried
once per second until the deadline. The cause label is still
`renewal_error`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, code search, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-04 10:15:08 -08:00
DottaandPaperclip 9ca9af544a Drive remaining live run and resource recovery from durable work (#18419)
Reviewed and verified at 6a4c193906.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-31 16:00:35 -05:00
bbcf5dfe98 fix(heartbeat): renew legacy controller leases on a private pool (#17032)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A legacy (non-native) agent run keeps a controller lease on its
`heartbeat_runs` row. The server renews the lease every 10 seconds. If
no renewal succeeds in 60 seconds, the watchdog stops the run with
`renewal_deadline`.
> - #16359 added timings to lease-loss reports. The first staging event
with these timings shows `deadlineLateMs=0`, `renewalPendingMs=50000`,
and 108 renewals in about 22.5 minutes, where about 135 are expected.
Thus the event loop was not blocked. One renewal UPDATE was in flight
for 50 seconds, and renewals were slow for a long time before the loss.
> - The agent had exited with code 0. The run was in post-execution
`sync_back`. Other runs on the same host completed. No code path held a
lock on the run row across slow work. A later `instruction_save` UPDATE
on the same row completed while the earlier renewal was still pending,
so a row-lock wait does not explain the stall.
> - postgres.js 3.4.9 sends a new query to a busy connection when no
connection is idle (`handler` → `go(busy.shift(), query)` in
`src/index.js`). The query waits behind the statement that is in flight
on that connection. The renewal had no timeout, and the `pending` flag
stopped all later renewals. Thus one slow unrelated statement on the
shared pool can stop a healthy run.
> - This pull request moves lease renewal to a small private pool,
bounds each attempt with a statement timeout, and retries a timed-out
attempt until the existing lease deadline.
> - The benefit is that slow application queries on the shared pool can
no longer cancel a healthy legacy run.

## Linked Issues or Issue Description

Refs #16359 (lease-loss timings that identified this cause)
Refs #16358 (pre-dispatch lease loss becomes retryable; different path)
Refs #16917 (open: keeps a clean adapter result after a lease lapsed; it
does not prevent the loss)
Refs #13389 (open: host sleep / late timers; different cause)

**What happened?**
A staging legacy `claude_local` run was cancelled with `Legacy
controller lease lost (renewal_deadline)` after the agent exited with
code 0. The timings show that the deadline timer ran on time, and one
renewal UPDATE was pending for 50 seconds.

**Expected behavior**
A live controller keeps its lease. Slow unrelated database work does not
cancel the run.

**Steps to reproduce**
1. Create a database client with one pooled connection.
2. Start `select pg_sleep(5)` on that client.
3. Call `watchLegacyControllerLease(...).assertOwned()` with the same
client.
4. Before this change, the renewal waits about 5 seconds behind
`pg_sleep`. With a long enough slow statement, the lease deadline stops
the run.

## What Changed

- `packages/db`: Add `livenessDb(db)`. It gives each `createDb` client a
private pool of at most 2 connections
(`application_name=paperclip-controller-lease`). The pool is created on
first use. Test doubles and transactions get the original client back.
- `packages/db`: Add `closeLivenessDb(db, timeoutSeconds)`. The server
calls it in `closeDatabaseClients`, so shutdown and failed startup end
the private pool too.
- `legacy-controller-lease.ts`: The watchdog renews in a transaction on
the private pool. Each attempt sets a transaction-local
`statement_timeout` of 10 seconds. A timed-out attempt (`57014`) is
retried. After a successful retry, the local deadline is measured from
that attempt's start, because that attempt's UPDATE wrote the fresh 60
second lease. The existing 60 second deadline, the abort race, and the
`renewal_rejected` / `renewal_error` attribution do not change.
`renewLegacyControllerLease` itself does not change, so the
runtime-selection caller inside its own transaction is not affected.
- Tests: a saturated shared pool no longer delays renewal; a renewal
that times out on a locked run row retries and keeps the lease; the
private pool is cached per client, has its own `application_name`, and
closes; after a 45 second timed-out attempt and a successful retry, the
next deadline is a full 60 seconds after the retry. The existing test
doubles now model the transaction.
- `docs/deploy/database.md`: Count the up to 2 extra connections per
server process when you size `max_connections`.

## Verification

- `cd server && npx vitest run
src/services/legacy-controller-lease.test.ts`: 20 passed, on embedded
Postgres.
- Red/green: with the old `legacy-controller-lease.ts` and the new
tests, the saturated-pool test fails with `expected 5018 to be less than
2000`. With the fix, it passes.
- Red/green for the retry deadline: with the deadline measured from the
first attempt, the retry-deadline test fails (`expected true to be
false`) because the run is aborted 15 seconds after the retry. With the
fix, it passes.
- Mutation check: when the timeout retry is removed, the locked-row test
fails with `canceling statement due to statement timeout`.
- `npx vitest run src/__tests__/heartbeat-process-recovery.test.ts -t
"lease|dispatch|controller|interaction"`: 43 passed.
- `npx vitest run src/services/heartbeat/run-completion.test.ts
src/__tests__/heartbeat-orphaned-active-lease-sweep.test.ts`: 53 passed.
- `npx vitest run src/__tests__/server-startup-feedback-export.test.ts`:
passed.
- `cd packages/db && npx vitest run src/work-signals.test.ts
src/client-options.test.ts src/client-teardown-registry.test.ts`: 24
passed. `npx tsc --noEmit` for `packages/db` passed.
- Not run locally: the full server `tsc` and the full server suite. The
sandbox has a 4 GB memory limit, and `tsc` was killed (exit 137). CI is
the gate for the server typecheck, the full tests, and the build.

## Risks

- Each server process can open up to 2 more Postgres connections.
Deployments that are at their `max_connections` limit must allow for
them. The docs now say this.
- A renewal attempt is now 4 statements (BEGIN, `set_config`, UPDATE,
COMMIT) instead of 1. Each statement is a short primary-key operation on
a dedicated connection.
- If many runs wait on locked run rows at the same time, renewals for
other runs can queue on the 2 private connections. The 10 second
statement timeout limits each wait, which is less than the 60 second
lease.
- Behavior for a real lease loss does not change. A rejected renewal and
a non-timeout error still stop the run at once, and the 60 second
deadline still applies.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code in agent mode with tool use (shell, file edit, and test execution).
A read-only subagent helped with code search.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 18:13:53 -07:00
ccc77c5b5e fix(server): record legacy controller lease-loss timings (#16359)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The heartbeat runner executes legacy agent runs, and a controller
lease watchdog keeps durable ownership of each run
> - The watchdog renews the lease every 10 seconds and stops the run
when a renewal does not finish within the 60-second lease
> - A staging run failed in setup with `LegacyControllerLeaseLostError`
and the `renewal_deadline` cause, after about eight minutes of
preparation and with no server restart
> - The setup failure path did not record the cause, and no evidence
showed why the renewal did not finish: a stalled database query or a
blocked event loop
> - This pull request records bounded watchdog timings with each lease
loss, on the setup failure path and on the cancellation path
> - The benefit is that the next event shows which of the two causes
stops renewal, which is the missing fact to fix the root cause

## Linked Issues or Issue Description

Refs #16084 (added the closed lease-loss cause for cancelled runs)
Refs #16355 (retries a run that a restart stopped before dispatch; a
different path)

**What happened?**
A legacy `paperclip_runner` run on staging failed with `setup_failed`
and the message "Legacy controller lease lost". The stack shows the
watchdog deadline timer (`renewal_deadline`). The run was in
`executionStage: "preparing"` for about 467 seconds. The server started
about 11 minutes before the run, so this was not a restart. The run
record and the Sentry event do not show the lease-loss cause, because
the setup failure path does not read it. Nothing shows if a renewal
query was in flight at the deadline, or if the deadline callback ran
late.

**Expected behavior**
When the watchdog loses the lease, the run records the closed cause and
the watchdog timings. This applies when the loss fails setup and when it
stops the adapter. The failure report contains only the closed cause and
bounded numbers.

**Steps to reproduce**
1. Start a legacy run with a controller lease.
2. During setup, make a lease renewal query hang for more than 60
seconds, or block the event loop past the deadline.
3. Let the watchdog abort the run.
4. Read `resultJson` on the failed run. Before this change, it has no
lease-loss evidence.

## What Changed

- `legacy-controller-lease.ts`: The watchdog tracks the last successful
renewal, the renewals in flight, the scheduled deadline, and the renewal
count. `LegacyControllerLeaseLostError` now carries `leaseLossTiming`:
`sinceRenewalMs`, `renewalPendingMs` (or `null`), `deadlineLateMs` (only
for `renewal_deadline`), and `renewalCount`. Ownership behavior does not
change.
- `run-cancellation.ts`: Add the `LegacyControllerLeaseLossTiming` type
and `readControllerLeaseLoss()`. The reader accepts only a closed cause
and non-negative finite numbers, and clamps each value to 24 hours.
- `heartbeat/run-completion.ts`: Persist
`resultJson.controllerLeaseLoss` with the cause and timings. This
applies to a lease loss that stops the adapter and to a lease loss that
fails setup (the thrown abort reason).
- `run-failure-diagnostics.ts`: Report `controllerLeaseLossCause` and
the four bounded timing fields from `controllerLeaseLoss`. A setup
failure now gets the cause label too.
- `doc/run-log-events.md`: Document the new record and how to read the
timings.
- Tests: watchdog timings for a stalled query, a blocked event loop, and
a failing query; completion records the evidence for an adapter stop and
for a setup failure; the failure report clamps values, drops unknown
keys, and drops a record without a closed cause.

## Verification

- `pnpm --filter @paperclipai/server exec vitest run
src/services/legacy-controller-lease.test.ts
src/services/heartbeat/run-completion.test.ts
src/services/__tests__/run-failure-report.test.ts
src/services/run-cancellation.test.ts`
- I could not run these tests locally. The sandbox has a 4 GB memory
limit, and `pnpm install` was stopped by the out-of-memory killer during
lockfile resolution. CI is the test run for this change.
- After deployment, a new lease loss shows
`controllerLeaseRenewalPendingMs` and `controllerLeaseDeadlineLateMs` in
the Sentry execution context. A pending time near 50 seconds or more
points at the database. A large deadline lateness points at the event
loop.

## Risks

- Low risk. This change adds diagnostics. It does not change when the
watchdog stops a run, and it does not prevent the lease loss.
- The new `resultJson.controllerLeaseLoss` field is server-written.
Readers ignore unknown values. Old rows do not have it.
- The failure report gets five closed or numeric fields. It gets no
reason text, IDs, or user content.
- The event-loop test blocks the loop for about 80 ms to make the
deadline late. It uses real timers, so a very slow CI host makes the
lateness larger, not smaller.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with
tool use (shell, file edits, Sentry and GitHub connectors). The exact
context-window size is not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [ ] I have run tests locally and they pass (not possible in this
sandbox; see Verification)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Devin Foley <139239+devinfoley@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 19:52:23 -07:00
Devin FoleyandClaude Opus 5.5 5e8956c207 fix(heartbeat): retry legacy runs a restart stopped before dispatch (#16355)
A hard server restart can stop a legacy run after it is claimed but before the adapter is dispatched. The next boot reaped it as failed/process_lost with no retry and held it for operator reconciliation, so the task wake was lost.

The legacy controller only writes executionStage "dispatching" in the lease renewal that admits the adapter. The orphan reaper now revokes the expired foreign-boot controller first, and uses the stage and process identifiers returned by that same UPDATE ... RETURNING. If the run is still "preparing" with no process identifiers, it records bootstrap evidence (providerWorkStarted: false) and uses the existing one-time process-loss retry. In "unexpected" mode, run-failure reporting does not send this retried restart interruption to Sentry. Runs that reached dispatching, native runs, and runs owned by the current boot are unchanged.

Verified exact head 91f1352097: full CI green, Greptile 5/5, no unresolved review threads, no conflicts. The tests were not run locally (sandbox memory limit); CI is the test run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 18:27:11 -07:00
a64969b288 fix(server): attribute legacy runs stopped by controller lease loss (#16084)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The heartbeat runner executes legacy agent runs. A controller lease
watchdog keeps durable ownership of each legacy run.
> - When the watchdog loses the lease, it aborts the adapter. The abort
reason was a plain `Error`, and nothing recorded why the run stopped.
> - Completion then saved the cancellation as source `unknown`. Sentry
got an unexpected cancel that it could not tell apart from other
unrecorded stops.
> - We saw this on a sandboxed Grok run. The system cancelled it after
about 22 minutes with `cancellationSource: unknown`. Workspace copy-back
then failed. Every `cancelRun` path records a stop request, so an
unrecorded stop this late points to the lease watchdog.
> - This pull request records a lease loss as its own cancellation
source, with a closed cause label.
> - The benefit is that the next event shows whether the lease loss
caused the stop, and which renewal failure caused it. That is the
missing fact to fix the root cause.

## Linked Issues or Issue Description

Refs #15790
Refs #15920

**What happened?**
A legacy sandbox run was stopped by the system while it was working. The
run record and the Sentry diagnostics show `cancellationSource:
unknown`, with no initiator. The controller lease watchdog aborts the
adapter with a plain `Error("Legacy controller lease lost")`. Completion
does not read the abort reason, so it records the stop as `unknown`.

**Expected behavior**
When the host stops its own adapter because it lost the controller
lease, the run records that cause. Diagnostics name the cause with a
closed label. A stop that a user or the control plane requested earlier
keeps its recorded source.

**Steps to reproduce**
1. Start a legacy run that has a controller lease.
2. Make lease renewal fail: the renewal query errors, the update matches
no row, or the lease deadline passes.
3. Let the adapter return its cancelled result.
4. Read `resultJson.cancellation` on the run. Before this change,
`source` is `unknown`.

**Agent adapter(s) involved**
Not adapter-specific (core bug). We saw it with `grok_local` in a
sandbox.

**Additional context**
This change adds diagnostics. It does not prevent the lease loss. #15920
retries a transient copy-back failure. It does not help here, because
the stopped sandbox rejects every retry.

## What Changed

- `legacy-controller-lease.ts`: The watchdog aborts with a new
`LegacyControllerLeaseLostError`. The error carries a closed cause:
`renewal_rejected`, `renewal_deadline`, or `renewal_error`.
- `run-cancellation.ts`: Add the `controller_lease` cancellation source.
Add `controllerLeaseLossCancellation()` and
`readControllerLeaseLossCause()`. The reader accepts only the
server-written reason text.
- `heartbeat/run-completion.ts`: When the outcome is `cancelled` and the
abort reason is a lease loss, record a `controller_lease` cancellation
(`expected: false`). This covers both an adapter that returns a
cancelled result (`completeRun`) and one that throws after the abort,
for example through `signal.throwIfAborted()` (`failRun`). A stop
request recorded earlier still takes precedence.
- `run-failure-diagnostics.ts`: Add `controllerLeaseLossCause` to the
execution diagnostics. Only the closed label goes to Sentry, never the
reason text.
- `heartbeat/run-state.ts`: preserve `controller_lease` in the bounded
SQL projection when saved results exceed 64 KB. A database regression
verifies the default run-detail read.
- `canContinueCancelledRun` stays unchanged. It does not accept
`controller_lease`, so a lease loss does not authorize automatic
continuation.
- `doc/run-log-events.md`: Document the new source and cause labels.
- Tests: unit tests for the cause on each watchdog path, completion
tests that record the source when the adapter returns and when it throws
the abort reason (both fail without the change), tests that a recorded
stop request takes precedence, and a test that the Sentry report
contains the closed label and not the reason text.

## Verification

- `cd server && npx vitest run src/services/run-cancellation.test.ts
src/services/legacy-controller-lease.test.ts
src/services/__tests__/run-failure-report.test.ts
src/services/heartbeat/run-completion.test.ts`: 238 passed, including
the embedded Postgres suites.
- I reverted only `run-completion.ts` and ran the new completion test.
It failed with `source: "unknown"`. With the change, it records
`controller_lease`.
- I removed only the `failRun` attribution and ran the new thrown-error
test. It failed without the `controller_lease` record, and passes with
it.
- Final full repository `pnpm -r typecheck` and `pnpm build` pass.

- Final local regression verification: all 247 tests pass across
run-state, cancellation, watchdog, failure reporting, and completion
suites. The large-result test fails before the SQL allowlist change. All
latest-head CI checks pass (54 successes and two intentional skips).
Fresh Apex is 5/5 on `708856c568d07d7b446a4a4af2edc6d9eec7d155` with no
unresolved findings.

## Risks

- Low risk. The change adds a new source value to server-owned JSON. Old
rows keep their values. The readers accept the new value, and no UI code
switches on this field.
- A lease-loss cancel stays `expected: false`, so it is still reported
as unexpected, as before. Now it has a more specific source.
- The Sentry payload gets one new closed label
(`controllerLeaseLossCause`). It does not include reason text or IDs.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with
tool use (shell, file edits, test runs).

OpenAI GPT-6 (`gpt-6`) through Codex added the large-result regression
and SQL projection fix, with shell, file-edit, and test tools. The exact
context-window size is not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-13 14:50:40 -07:00
Nicky LeachandPaperclip dbf5ea432d fix: protect starting runs during overlapping deployments (#13285)
## Thinking Path

> - Paperclip controls agent work across service deployments.
> - A run can provision a remote sandbox before a process or invocation
event exists.
> - Each container previously treated its own missing process handle as
proof that the run was orphaned.
> - Overlapping deployments could therefore fail a run owned by another
container.
> - This pull request records and renews a controller lease before
provisioning.
> - A recovery worker must revoke an expired owner before it finalizes
the run.

## Linked Issues or Issue Description

Merged PR #13272 records startup adapter identity and restores explicit
user continuation. This PR adds controller ownership on top of current
master. Refs #7997 and #10442 for related replica and ownership
problems. Related #13138 addresses silence and detached local processes;
this change does not infer death from silence.

**What happened?**
During an overlapping hosted service deployment, a new container reaped
a legacy conversation run that another container was provisioning. The
run had no PID or adapter invocation yet.

**Expected behavior**
A live controller keeps its run. After controller loss, one recovery
worker takes cleanup authority and the old controller cannot dispatch
further work.

**Steps to reproduce**
Claim a legacy run in controller A. Start controller B against the same
database before A finishes provisioning. Run the startup reaper in B.

**Paperclip version or commit**
Observed on `663c44cb2b9c28336d38d0b4a6971f4f1964bce6` in a hosted
Railway deployment with a Daytona environment.

## What Changed

- Add nullable controller boot ID, lease deadline, and execution stage
columns. Claim ownership in the queued-to-running update.
- Renew ownership independently of run output. Abort and reject dispatch
if renewal fails.
- Serialize reaper revocation against renewal. Let unfinished recovery
claims expire after a restart.
- Restrict graceful shutdown to legacy runs owned by the current
controller.
- Hand ownership back to the existing native coordinator when runtime
selection becomes native.
- Add twelve database regressions and document the lease contract.
Update the task-drain regression to require controller expiry before
reaping.

## Verification

- `pnpm exec vitest run
server/src/services/legacy-controller-lease.test.ts
server/src/__tests__/heartbeat-task-drain-admission-release.test.ts`: 14
passed after rebasing onto master (`f12b647ae`).
- Queue-interruption regressions in
`heartbeat-process-recovery.test.ts`: 2 passed after preserving the new
cleanup promotion from #13275.
- `pnpm --filter @paperclipai/server exec tsc --noEmit`: passed after
rebuilding runner TypeScript outputs for the updated master. Broad local
tests are omitted at the maintainer’s request; CI owns broad coverage.

- Latest-head CI passed on `f255e8e4d5ab2b24b12638a02434e6aa8a2285c5`:
[run
34658248569](https://github.com/paperclipai/paperclip/actions/runs/34658248569).
All test shards, browser suites, typecheck, build, canary, and security
checks passed. Greptile is 5/5 with no unresolved review threads.

## Risks

- Additive, idempotent migration; historical rows retain the previous
recovery behavior.
- Database unavailability aborts new dispatch rather than permitting an
unfenced controller to continue.
- Lease expiry is permission to clean up, not evidence that remote
inference stopped. Follow-up PRs add persistent cleanup and automatic
continuation.
- Mixed-version deployment still includes old binaries whose reapers do
not understand controller leases.

## Model Used

OpenAI GPT-6 through Codex, using reasoning, repository inspection, code
execution, and test tools. The precise backend revision and
context-window size are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-09-11 16:55:44 -07:00