mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-09 07:10:34 +01:00
Nicky LeachandClaude Opus 5.5
026dc6d135
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs keep a finalization row. Its lease owner tells the
control plane that an execution may still be live.
> - Pausing an agent waits for `agentExecutionsHaveStopped`. That check
refuses to finish while any run of the agent has a lease owner, unless
the run was cancelled and has stop proof.
> - A same-run session resume claims that lease before it dispatches the
run. If the run then fails before the native executor takes its own
provider claim, nothing releases the resume owner.
> - The leaked owner blocks pause and resume for good, and the task
keeps looking like it has a live recovery path.
> - This pull request settles those rows at the source, and it adds a
stop-proof-gated release for rows that already leaked.
> - The benefit is that agents can pause and resume after an ordinary
resume failure, without database edits, and the cancelled-run stop-proof
rule stays as it is.
## Linked Issues or Issue Description
**What happened?**
An agent stayed in `pausing` with `hostComplete=false` and climbing
attempts. Its runs had `native_run_finalizations` rows with `phase:
observed`, `recovery_state: resuming_session`, a `lease_owner` like
`default:resume:<uuid>`, and no `result_id`. Each heartbeat run had
ended `failed`: either `setup_failed` (`Failed to acquire lease for
environment … Sandbox creation has failed`,
`native_remote_recovery_lease_mismatch`) or `adapter_failed`
(`runner_state_identity_mismatch: …`). The lease expired about 20
minutes after the run finished. Nothing settled the row. The only fix
was to set `lease_owner` to null by hand.
**Expected behavior**
When a resumed run fails before any provider turn starts, its
finalization row goes to `terminal_failure` and its lease owner is
released, as the native executor does for failures it sees. A row that
leaked before this fix is released in the background once its lease
expires and the run has stop proof. Pause then completes.
**Steps to reproduce**
1. Leave a native run in `retryable_failure` with a persisted
checkpoint, on a task that already has an active recovery incident.
2. Let session resume claim it (`claimNativeSessionResumptions` sets
`phase: observed` and a `…:resume:…` owner).
3. Make the resumed run fail before the executor claims it, for example
when the environment driver cannot create the sandbox.
4. The heartbeat failure path marks the run `failed`. The issue release
path does not settle the row, because it settles only when it opens a
new incident. The owner stays set, and `agentExecutionsHaveStopped`
returns false indefinitely.
**Paperclip version or commit**
`master` at 5ac05871b5.
**Additional context**
Root cause: two writers can end a resumed run without the executor.
`failRun` and the setup-failure path in `heartbeat/run-completion.ts`
never touch the coordinator. `recordNativeTerminalRecoveryIfNeeded` in
the wake-queue adapter settles the coordinator only when no recovery
incident exists yet. Agents that need their computer connection paired
again usually already have an incident, so the owner leaks.
Related, but different: #18905, #18866, #18072, and #17692 change how
the stop check judges runs. This pull request does not change the stop
check.
## What Changed
-
`server/src/services/native-runtime/native-failed-run-owner-release.ts`
(new): one settlement for a result-less `observed` owner on a run that
ended `failed`, `timed_out`, or `interrupted`. It writes
`terminal_failure`, null lease fields, `recoveryState: "blocked"`, the
run's failure code, and a `releasedOwner` record. It sets `nativePhase`,
sends the same finalization and execution work notifications as the
executor's terminal path, and logs `native.failed_run_owner_released`
activity. Locks follow the issue, coordinator, run order.
- `settleUnstartedNativeSessionResume`: runs after a dispatched session
resume returns. It matches only the exact claim owner and the attempt
the claim saw. The executor increments the attempt when it takes
provider ownership, so an unchanged attempt proves that no provider turn
started under this owner. The executor's own failures no longer match.
- `releaseAbandonedNativeOwners`: the background release for rows that
already leaked. It requires an expired lease and all of the stop proof
that `agentExecutionsHaveStopped` checks: settled environment leases, no
live run process or process group, no live controller lease, and no live
controller process. This process answers from its in-memory execution
registries. It skips cancelled runs, pending native Stops, and pinned
workspace restores, because those have their own stop-proof owners.
- `native-finalization-reconciler.ts`: `NativeSessionResumeClaim` now
carries `attempt`.
- `heartbeat.ts` and `heartbeat/recovery.ts`: both session-resume
dispatch sites settle the claim after `executeRun` returns. This is one
`.finally` per site.
- `execution-control-reconciliation.ts` and `execution-control-work.ts`:
the execution-finalization queue runs the background release. Its
deadline includes the earliest lease expiry among eligible rows, so the
release does not poll an idle instance. Rows with unreleased sandboxes
are excluded from the deadline.
- `agent-execution-stop.ts` is unchanged. The cancelled-only release
keeps its stop-proof rule.
## Verification
- `npx vitest run
src/__tests__/heartbeat-native-resume-lease-failure.test.ts` drives the
real heartbeat resume dispatch (`reapOrphanedRuns`). A sandbox driver's
`acquireRunLease` throws `Sandbox creation has failed`, and the task has
a recovery incident. The run fails with `Failed to acquire lease for
environment "Managed computer" (sandbox): Sandbox creation has failed`,
and the coordinator ends `terminal_failure` with no lease owner. I
removed the `.finally` wiring and ran the test again. It then failed,
and the row stayed `observed` with a `default:resume:` owner and a
20-minute expiry, which is the production state.
- `npx vitest run
src/services/native-runtime/native-failed-run-owner-release.test.ts` (10
tests):
- A claimed resume that fails at lease acquisition is settled.
`agentExecutionsHaveStopped` changes from false to true. A repeated
settlement does nothing.
- An owner the executor claimed (incremented attempt) and a run that is
still running are not changed.
- A leaked `runner_state_identity_mismatch` row is released by
`reconcileAbandonedExecutionControl`, and then
`agentExecutionsHaveStopped` returns true.
- An unexpired owner still blocks the stop check. Its expiry becomes the
finalization queue deadline.
- An active environment lease and a live controller process block
release. After both stop, the row is released.
- Cancelled runs and pending native Stops are not changed.
- Added during review:
- Releasing an owner wakes the cost-accounting worker.
- Releasing a run's environment lease wakes the finalization worker, and
the owner is then released.
- A recorded PID and group that now belong to an unrelated process do
not hold the owner.
- Bounded passes rotate past a held row (`[0, 1, 1]` with page size 1),
then wrap back to it.
- The immediate settle keeps the owner while the run's recorded process
is alive.
- Suites that cover the changed files pass together: the two new files,
`native-session-resumption.test.ts` (with the claim shape updated),
`execution-control-reconciliation.test.ts`,
`execution-control-work.test.ts`, and
`agent-lifecycle/__tests__/lifecycle.test.ts`. With
`environment-cleanup-work.test.ts` added, that is 7 files and 156 tests.
I did not run the full `heartbeat-process-recovery.test.ts` suite
locally. CI runs it.
- `pnpm --filter @paperclipai/server typecheck` passes.
## Risks
- Behaviour change: a resumed run that fails during setup now ends with
a terminal coordinator in every case. Before, this happened only when
the issue release path opened a new incident. After a restart, restart
recovery no longer sees such a row as `observed`. This matches what the
release path already does when no incident exists.
- The background release trusts an expired lease on a run that is
already terminal, together with the same process and lease evidence that
the stop check uses. A controller on another host cannot be probed by
PID. The stop check has the same limit today. A live owner still renews
its lease, and lease-gated writes from an expired owner are already
refused.
- If an expired row's process or controller still looks live, the
release retries at the shared 15-second finalization cadence until that
process exits. Each pass handles one bounded page, and a rotating cursor
moves to the next page on each pass.
- A run-scoped environment cleanup notification now also wakes the
execution-finalization queue. Passes are coalesced, and a pass is a
cheap deadline query when nothing is eligible.
- No schema change and no migration.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`) through Claude Code, with tool use
(shell, file edits, local test runs).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
72 lines
5.2 KiB
Markdown
72 lines
5.2 KiB
Markdown
# Execution-control workers and idle instances
|
|
|
|
Execution-control maintenance uses the shared delivery coordinator instead of an
|
|
unconditional 15-second interval. This changes when recovery is inspected; it
|
|
does not change the authority, evidence, locking, idempotency, or cleanup rules
|
|
inside the six owning services. No table or migration is added.
|
|
|
|
## Duties and durable state
|
|
|
|
| Duty | Trigger and durable source | Remaining scheduled work | Quiet condition |
|
|
| --- | --- | --- | --- |
|
|
| Abandoned finalization | Committed control-deadline writes on `heartbeat_runs` and `native_run_finalizations`; committed run transitions that leave a result-less `observed` owner on a failed, timed-out, or interrupted native run | Earliest non-null control deadline, or earliest lease expiry of such an owner whose environment leases are settled; a due row still present after recovery retries after 15 seconds | No control deadlines and no such owner |
|
|
| Safe native replacement | Committed run transitions and reconciliation decisions; existing failed run/coordinator rows | Retry while the existing replacement predicate selects a candidate, including one awaiting provider termination or sandbox release | No replacement candidate, or a recorded successor/denial |
|
|
| Reconciled continuation | Committed recovery-action marking/resolution; `evidence.continuationDelivery` | Retry pending delivery after 15 seconds | No resolved action awaiting continuation delivery |
|
|
| Execution status | Committed run status writes; `execution_status_delivery_id` | Retry while a delivery marker remains | All markers acknowledged |
|
|
| Automatic disposition | Committed run/recovery-action changes; existing eligible recovery-action queries | Retry while a selected action still needs disposition or an obsolete conversation hold remains | No eligible action/hold; resolved history alone is quiet |
|
|
| Local login cleanup | Committed local login creation; existing `adapter_auth_sessions.expires_at` | Earliest local login expiry; retry overdue failed cleanup | No local login expiry remains |
|
|
|
|
Each worker runs once on startup to restore durable obligations. Passes are
|
|
coalesced at the former 15-second resolution: a tenant's repeated writes cannot
|
|
force continuous global scans. This is a minimum spacing between requested
|
|
passes, not a recurring poll; an empty worker never arms that spacing by itself.
|
|
A newly discovered deadline within the spacing window runs at its end, retaining
|
|
the former maximum 15-second scheduling resolution. Future deadlines outside
|
|
that window retain their exact due time. Workers remain
|
|
independent: a slow replacement proof cannot stall status delivery. Batches and
|
|
failed obligations retain the existing 15-second retry budget; empty workers do
|
|
not repeatedly inspect the database. Login authorization still checks expiry on
|
|
use even before physical cleanup completes.
|
|
|
|
## Producer contract
|
|
|
|
`notifyExecutionRunWork(transaction)` signals the run transition's status,
|
|
finalization, replacement, and disposition consumers before the write. Existing
|
|
transaction owners call it directly. Standalone deadline and recovery-action
|
|
writes use `writeExecutionControl` to give the write a transaction boundary.
|
|
Nested transactions retain the existing outer-commit notification semantics.
|
|
|
|
Producer coverage includes heartbeat status transitions, legacy terminalization,
|
|
native finalization and restart/failure settlement, replacement successors and
|
|
denials, recovery-action upserts/resolution, operator reconciliation, control
|
|
deadline creation/cancellation, and local login creation. Workers acknowledge
|
|
existing records rather than creating a second queue. The replacement and
|
|
disposition pending checks reuse the executing services' eligibility queries.
|
|
|
|
This is the existing single live server / shared `createDb` owner signal model.
|
|
Independent SQL writers or separate replicas must integrate notifications; these
|
|
workers do not continuously discover unannounced external writes. Startup still
|
|
recovers committed rows after process loss. Uncertain commit handling remains in
|
|
`packages/db/src/work-signals.ts`.
|
|
|
|
## Lifecycle and verification
|
|
|
|
A worker consumes a requested generation only after its sweep and deadline
|
|
inspection succeed. Writes during a sweep, startup during drain, and a failure
|
|
before drain remain pending for resume. Passive waiting during drain performs no
|
|
worker SQL. Shutdown uses the coordinator to unsubscribe and await in-flight work.
|
|
|
|
`server/src/__tests__/execution-control-work.test.ts` covers empty and retained
|
|
history with the real sweep implementations, a real producer inside an outer
|
|
commit/rollback, saved deadlines, deadline replacement/cancellation, durable
|
|
status retries, recovery continuations, independent duties, and drain/resume.
|
|
Database tests use embedded PostgreSQL by default. An isolated external test
|
|
instance can be supplied through `PAPERCLIP_EXECUTION_SLEEP_TEST_DATABASE_URL`;
|
|
the database must have a dedicated name ending in `_test` and is cleared by the
|
|
suite. Never use a tenant database.
|
|
|
|
Worker quiet is not host sleep authorization. The existing idle-safety blockers
|
|
remain, and Cloud's external wake contract does not gain coverage for execution
|
|
or login deadlines through this change. Provider suspension and wake must be
|
|
qualified separately before relaxing those blockers.
|