mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-13 07:12:00 +01:00
DottaandClaude Opus 5.5
148d2b629c
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A legacy run keeps a controller lease on its `heartbeat_runs` row.
The server renews every lease in the process with one batched statement
every 10 seconds, and stops a run when its local lease deadline passes.
> - The batched renewal ran on the 2-connection liveness pool. Native
runner lease renewals use the same pool. Each native renewal holds a
connection for a row lock, about seven notifications, and an update.
> - With 18 native runner runs renewing every 20 seconds on a slow
database, the legacy batch waited 30 to 50 seconds for a connection. The
local deadline also counted that wait, so it fired 20 to 40 seconds
before the database lease could expire.
> - Runs then stopped with `controller_lease_lost` while no other
controller owned them, and their issues looped through retries.
> - This pull request gives legacy renewal its own connection and
measures the deadline from the time the statement starts on that
connection.
> - The benefit is that a slow server no longer stops runs that still
own their lease. A real takeover still stops the run.
## Linked Issues or Issue Description
**What happened?**
On staging, after the 05:37Z deploy, 63 of about 175 new runs failed
with `controller_lease_lost` ("Legacy controller lease lost") at about 3
per minute. All were legacy runs in preparation
(`executionRecovery.kind: bootstrap`, `providerWorkStarted: false`). One
example had `controllerLeaseLoss: {cause: "renewal_deadline",
renewalCount: 7, sinceRenewalMs: 42537, renewalPendingMs: 38798,
deadlineLateMs: 134}`. Several runs that failed at the same moment had
identical timing values, so they were members of the same renewal batch.
The previous successful renewal took 17 to 44 seconds, so the deadline
that it set was only 16 to 43 seconds away. At that time 27 legacy runs
were preparing and 18 native runner runs were renewing on the same
liveness pool. The issues looped: `issue_assigned`, lease lost,
`issue_continuation_needed`, lease lost, `transient_failure_retry`,
lease lost.
**Expected behavior**
A slow server or database must not cause a lease loss while the
controller still owns the lease. Only a real expiry or a takeover by
another controller must stop the run.
**Steps to reproduce**
1. Start legacy runs and native runner runs on one server.
2. Make the database slow, so that each native lease renewal holds a
liveness connection for several seconds.
3. The legacy renewal batch waits for a liveness connection, and its
next local deadline is shortened by that wait.
4. The next renewal also waits, and the local deadline fires first. The
run fails with `controller_lease_lost` (`renewal_deadline`).
**Paperclip version or commit**
Staging at 8cefbe9977. The fix is based on master at aa94939988.
**Deployment mode**
Hosted cloud (staging).
**Agent adapter(s) involved**
Legacy controller runs (Codex and Claude local adapters). Native runner
runs share the liveness pool.
## What Changed
- `packages/db`: add `controllerLeaseDb(db)`, a private 1-connection
pool (`paperclip-legacy-controller-lease`) for legacy controller
renewal. `closeLivenessDb` also ends it, so shutdown does not change.
- `legacy-controller-lease.ts`: the batched renewal runs on
`controllerLeaseDb` instead of the liveness pool that native lease
renewals use.
- `legacy-controller-lease.ts`: each batch records the time when its
transaction starts on the connection. The watcher measures the next
deadline from that time, not from the time the request was queued. The
database sets the new expiry at UPDATE time, which is later, so the
local deadline never passes the database expiry.
- The renewal statement, its boot-id and expiry conditions, and
revocation do not change. A takeover or an expired lease still rejects
the renewal and stops the run.
## Verification
- Red/green regression tests in
`server/src/services/legacy-controller-lease.test.ts`:
- `renews without queueing behind native lease transactions that hold
every liveness connection` (embedded Postgres). Before the fix, the
renewal waited 9.8 s behind four 5-second liveness transactions. After
the fix, it completes in less than 2 s.
- `keeps a controller owned when every renewal waits thirty-five seconds
for a connection before its statement runs`. Before the fix, the lease
was lost with `{sinceRenewalMs: 25000, renewalPendingMs: 20000,
renewalCount: 1}`, the same pattern as on staging. After the fix, the
controller stays owned for four lease periods.
- Updated the private-pool tests to check the new pool and its
application name.
- `npx vitest run src/services/legacy-controller-lease.test.ts`: 40
passed.
- `pnpm verify` (affected tests), plus `typecheck` for `packages/db` and
`server`.
## Risks
- Each server process opens one more database connection, only when it
has a legacy run.
- The local deadline can now be later than before, by the time that the
batch waited for a connection. It stays at or before the database
expiry, because the database sets the expiry after the recorded start
time. Readers that act on an expired lease are not affected.
- Low risk otherwise. There is no schema change.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use and code execution.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
Refs #20924, #20936
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>